AFIT/GOR/ENS/99M- 1 8 


TECHNOLOGY  SELECTION  FOR  THE  AIR  FORCE 
RESEARCH  LABORATORY  AIR  VEHICLES 
DIRECTORATE:  AN  ANALYSIS  USING  VALUE 
FOCUSED  THINKING 
Michael  F.  Winthrop,  Captain,  USAF 
AFIT/GOR/ENS/99M- 1 8 


Approved  for  Public  Release;  Distribution  Unlimited. 


m  (gtfAUonr  immoTmp*, 


19990409  026 


The  views  expressed  in  this  thesis  are  those  of  the  author(s)  arid  do  not  reflect  the  official 
policy  or  position  of  the  Department  of  Defense,  the  Air  Force  Research  Laboratory,  or 
the  U.S.  Government. 


AFIT/GOR/ENS/99M- 1 8 


TECHNOLOGY  SELECTION  FOR  THE  AIR  FORCE  RESEARCH  LABORATORY 
AIR  VEHICLES  DIRECTORATE:  AN  ANALYSIS  USING  VALUE  FOCUSED 

THINKING 

Presented  to  the  Faculty  of  the  Graduate  School  of  Engineering  of  the  Air  Force  Institute 

of  Technology 
Air  University 

Air  Education  and  Training  Command 
In  Partial  Fulfillment  of  the  Requirements  for  the 
Degree  of  Master  in  Operations  Research 

Michael  F.  Winthrop,  B.S. 

Captain,  USAF 

March  1999 


Approved  for  Public  Release;  Distribution  Unlimited. 


AFIT/GOR/ENS/99M- 1 8 


TECHNOLOGY  SELECTION  FOR  THE  AIR  FORCE  RESEARCH  LABORATORY 
AIR  VEHICLES  DIRECTORATE:  AN  ANALYSIS  USING  VALUE  FOCUSED 

THINKING 


Michael  F.  Winthrop,  B.S.,  M.S, 
Captain,  USAF 
March  12,  1999 


Approved: 


Dr.  Richard  F.  Deckro,  Co-Chairman 
Professor  of  Operations  Research 
Department  of  Operational  Sciences 

_ 

LTC  Jack  M.  Kloeber^.,  Co-Chairman 
Assistant  Professor  of  Operations  Research 
Department  of  Operational  Sciences 


/L  79 

Date 


IAZZloL.  ?? 

Date 


Acknowledgments 


Thanks  go  to  the  Lord  Jesus  Christ  who  gave  me  the  strength  to  endure  the  many 
hours  this  effort  took.  Further,  I  would  like  to  thank  my  wife  Barbara  and  my  two 
children  Phillip  and  Kaitlyn.  I  love  all  of  them  and  they  have  had  to  deal  with  too  many 
days  and  nights  of  me  being  away  at  school  working  on  this  project. 

Thanks  also  go  to  Dr.  Dick  Deckro  and  LTC  Jack  Kloeber.  Dr.  Deckro  deserves 
credit  for  sharing  his  knowledge  of  research  and  development  (R&D).  He  also  read 
numerous  drafts  of  this  thesis,  provided  numerous  insights  of  what  is  important  in  R&D 
and  is  responsible  for  much  of  what  I  have  learned  about  R&D.  LTC  Kloeber  has  taught 
me  about  decision  analysis,  prevented  me  from  going  in  the  wrong  direction  many  times, 
and  has  always  been  an  encourager  with  his  constant  “can  do”  attitude. 

The  people  in  the  Air  Vehicles  Directorate  of  the  Air  Force  Research  Laboratory 
(AFRL)  contributed  large  amounts  of  their  time  providing  their  preferences,  expert 
knowledge  and  understanding  of  technology,  which  was  critical  to  this  study.  Mr.  Tim 
Dues  and  Mr.  Terry  Neighbor  provided  their  support  needed  to  begin  this  thesis.  Col. 
Wood,  the  Director  of  Air  Vehicles,  released  the  needed  resources  from  his  Directorate. 
Col.  Wood,  Dr.  Paul  and  Dr.  Borger  all  contributed  expertise  in  the  future  directions  Air 
Vehicles  is  moving  in. 

In  developing  the  Air  Vehicles  value  model,  Mr.  Richard  Colclough  provided  his 
insights  from  over  30  years  of  research  and  development  experience,  identified  key 
experts  and  set  up  numerous  meetings  with  Air  Vehicles  experts.  This  thesis  could  not 
have  been  completed  without  him.  Mr.  Colclough  attended  nearly  every  meeting  and 
never  tired  of  helping  me  in  my  effort. 


11 


The  integrated  concept  leaders:  Mr.  Albert  Boudreau,  Dr.  Yvette  Weber  and  Mr. 
Mike  Ziegler  led  by  Mr.  Brian  Van  Vleet  provided  crucial  insights  and  knowledge  for 
this  effort.  Dr.  David  Moorhouse,  Lt  Col  Docken,  Rick  Peters  and  Jim  Pruner  all  made 
various  levels  of  contributions  as  part  of  the  Strategic  Integrated  Process  Team,  lead  by 
Mr.  Colclough.  Others  who  provided  their  insights  are  Dennis  Sedlock,  Mart  Entz,  Ralph 
Speelman,  Bruce  Clough,  Maj  Jeff  Cole,  Ray  Robertas,  and  J.  B.  Schroeder. 

Scoring  of  actual  Air  Vehicles  programs  was  completed  with  the  expert  help  of 
Forrest  Sandow,  Jim  McDowell,  Doug  Dolvin,  Dave  Homan  and  Harry  Karasopoulos. 
Scoring  meetings  were  held  on  very  short  notice,  and  these  people  willingly  cleared  their 
schedules  to  help.  Additionally,  thanks  go  to  Bill  Baron  and  Russ  Osborne. 


iii 


Table  of  Contents 


Acknowledgments . ii 

List  of  Figures . viii 

List  of  Tables . xii 

Abstract . xiii 

Chapter  1:  Introduction . 1 

Background . 1 

Purpose . 3 

Scope . 4 

Summary  and  Organization . 5 

Chapter  2:  Literature  Review . 7 

Introduction . 7 

Aerospace  Research  and  Development  (R&D) . 7 

Definitions . 7 

Dimensions  Specific  to  Basic  Research . 9 

Dimensions  Specific  to  Applied  and  Development  Research . 10 

Dimensions  General  to  Basic,  Applied  and  Development  Research . 11 

Technological  Change . 12 

Dimensions  Specific  to  U.S.  Air  Force  and  Aerospace  R&D . 13 

Summary . 16 

Technology  Selection  Models . 18 

Examples  of  Government  Multidimensional  Scoring  Studies . 20 

Value  Focused  Thinking . 22 

Fundamental  Objectives  Hierarchy  or  Value  Hierarchy  Theory . 22 

Value  Hierarchy  Development  Methods . 23 

Measurement  Theory . 24 

Measurement  Scales  and  Single  Dimension  Value  Functions . 26 

Finding  Alternatives  and  Attribute  Scoring . 28 

Assessing  Weights . 29 

Multiobjective  Additive  Value  Functions . 30 

Ranking  of  Results . 32 

Sensitivity  Analysis . 32 

Resource  Allocation . 33 

R&D  Model  Characteristics  and  Known  Limitations . 36 

Literature  Review  Summary . 39 

Chapter  3:  Methodology  and  Modeling . 40 

Introduction . 40 

Air  Vehicles’  Model  Development  Overview . 41 

Model  Selection . 41 


IV 


Analysis  Process . 43 

The  Search  for  the  Air  Vehicles’  Value  Hierarchy . 45 

The  Method  for  Identifying  the  Hierarchy . 45 

Results  from  the  Literature . 46 

The  Air  Vehicles  Hierarchy . 48 

Developing  the  Air  Vehicles’  Value  Model . 52 

Measuring  Change  in  Support  to  the  Warfighter  due  to  Technology . 52 

Trading  off  Attributes  in  Favor  of  Others . 53 

Elicitation  of  Single  Dimension  Value  Functions . 55 

Using  the  Additive  Value  Function . 61 

Weighting  of  Values  using  a  Hypothetical  Example . 62 

Air  Vehicles’  Weights . 65 

Modeling  Assumptions . 68 

Specific  to  R&D  and  Air  Vehicles . 68 

Specific  to  the  Value  Model  and  the  Resource  Allocation  Models . 69 

Using  the  Air  Vehicles  Value  Model . 70 

Alternatives . 70 

Program  Evaluation . 72 

Sensitivity  Analysis . 72 

Scoring  Lessons  Learned . 73 

Suitability  of  the  Air  Vehicles’  Model . 74 

Methodology  and  Modeling  Summary . 80 

Chapter  4:  Results  and  Analysis . 81 

Introduction..... . 81 

Value  of  Air  Vehicles  Technology . 81 

Ranking  of  a  subset  of  Air  Vehicles’  Programs . 81 

Identifying  Technological  Gaps  using  the  Value  Model . 82 

Sensitivity  of  the  Air  Vehicles’  Value  Model . 86 

Analysis  of  “Support  the  Warfighter” . 87 

Analysis  of  Extremes  of  Technological  Superiority . 90 

Analysis  of  Gain  versus  Sustain  under  Reach . 90 

Space  versus  UAV and  Sustainment  Efforts  under  Reach/Gain . 92 

Critical  Uncertainties . 93 

The  Cost  of  Creating  Change  in  Warfighter  Technological  Support . 95 

Measuring  the  Change  in  Warfighter  Support  for  a  Program . 95 

The  Change  in  Warfighter  Support  due  to  Technology  versus  Program  Cost 

Assuming  “All  or  Nothing  ”  Funding . 99 

Change  in  Warfighter  Technological  Support  versus  Program  Cost  Assumes  Partial 

Funding  of  Programs . 103 

Other  Costs  Besides  Investment  Dollars . 107 

Summary . 108 

Chapter  5:  Findings  and  Conclusions . 110 

Introduction . 110 

Conclusions . 1 1 1 

Recommendations . 112 


Future  Research . 112 

Summary . 115 

Appendix  A:  USAF  Strategic  Documents  Literature  Search  Results . 116 

Appendix  B:  Documentation  of  Meetings  with  VA . 144 

VA-AFIT  Value  Hierarchy  Meeting  Highlights/4  Dec  98 . 144 

VA-AFIT  Value  Hierarchy  Meeting  Highlights/8  Dec  98 . 145 

VA-AFIT  Value  Hierarchy  Meeting  Highlights/14  Dec  98 . 147 

VA-AFIT  Value  Hierarchy  Meeting  Highlights/17  Dec  98 . 148 

VA-AFIT  Value  Hierarchy  Meeting  Highlights/21  Dec  98 . 150 

VA-AFIT  Value  Hierarchy  Meeting  Highlights/7  Jan  99 . 151 

VA-AFIT  Value  Hierarchy  Meeting  Highlights/19  Jan  99 . 153 

VA-AFIT  Value  Hierarchy  Meeting  Highlights/20  Jan  99 . 154 

VAA:  L/D  Evaluation  Measure/25  Jan  99 . 156 

VAC-AFIT  Reach  Measures/27  Jan  99 . 158 

VA-AFIT  Reach,  Power,  Tech.  Sup.  Measures/29  Jan  99 . 159 

VA  Directorate- AFIT  Meeting  Highlights/1  Feb  99 . 161 

VAS-AFIT  Weight  Reduction  Measure/2  Feb  99 . 162 

VAC-AFIT  Reach  Measures/2  Feb  99 . 164 

VA-AFIT  Reach,  Power,  Tech.  Sup.  Measures/4  Feb  99 . 166 

VA  Directorate- AFIT  Weight  Elicitation  Summary/4  Feb  99 . 167 

VA-AFIT  Measures,  Preferential  Independence,  Weights/10  Feb  99 . 170 

VA-AFIT  Concerns  over  some  Measures  and  Weights/19  Feb  99 . 172 

VA-AFIT  Scoring/22  Feb  99,  24  Feb  99  -  25  Feb  99 . 173 

Appendix  C:  Evaluation  Measure  Templates  and  Macros . 183 

Evaluation  Measure  Types . 183 

Excel  Button  Macros . 187 

Excel  Value  Function  Implementations . 190 

Macros  for  Integer  and  Linear  Programs  in  Excel . 192 

Appendix  D:  S-Curves  as  a  Value  Measure . 195 

Appendix  E:  Value  Hierarchies  and  Means  Networks . 197 

Appendix  F:  Air  Vehicles’  Evaluation  Measures . 199 

Appendix  G:  Air  Vehicles’  Weight  Calculations . 228 

Appendix  H:  Air  Vehicles’  Programs  and  Scoring . 237 

Description  of  Air  Vehicles  Programs  Scored . 237 

Tabulated  Raw  Scores . 239 

Appendix  I:  Graphical  Sub-value  Contribution  to  each  Value . 242 

Appendix  J:  Sensitivity  Analysis  of  all  Values  and  Evaluation  Measures . 248 


vi 


Appendix  K:  Intensity  of  Change  in  Warfighter  Support  due  to  Technology . 272 

Bibliography . 273 

Vita . 278 


vii 


List  of  Figures 


Figure  1 :  Study  Framework . 43 

Figure  2:  Top  Level  Value  Model  for  Air  Vehicles . 49 

Figure  3:  Value  Model  Breakdown  of  Reach,  Awareness  and  Power . 50 

Figure  4:  Value  Model  Breakdown  of  Technological  Superiority . 50 

Figure  5:  The  Air  Vehicles’  Value  Hierarchy . 5 1 

Figure  6:  Gain  Availability  Evaluation  Measure . 57 

Figure  7:  Sustain  Availability  Evaluation  Measure . 58 

Figure  8:  User  Rating  under  Technological  Superiority  under  Be  Efficient . 58 

Figure  9:  Air  Vehicles  Value  Model . 60 

Figure  10:  Difference  between  Global  and  Local  Weights . 63 

Figure  11:  Value  Contributions  for  “Support  the  Warfighter” . 85 

Figure  12:  Value  Contributions  to  “Reach” . 85 

Figure  13:  Value  Contributions  to  “Sustain” . 86 

Figure  14:  Sensitivity  Analysis  Key . 87 

Figure  15:  Sensitivity  of  Weight  for  Reach . 88 

Figure  16:  Sensitivity  of  Weight  for  Technological  Superiority . 90 

Figure  17:  Sensitivity  of  Weight  on  Gain  under  Reach . 91 

Figure  18:  The  Change  in  Warfighter  Benefit  versus  the  Cost . 97 

Figure  19:  Total  Change  in  Warfighter  Technological  Support  versus  Cumulative  Cost 
Results . 99 

Figure  20:  Optimized  Change  in  Warfighter  Technological  Support  versus  Investment 
Cost  using  0-1  Math  Program . 101 

Figure  21:  Portfolio  Mix  of  Programs  for  various  Investment  Costs  by  Cutting  Programs 
. 102 

Figure  22:  Optimized  Change  in  Warfighter  Technological  Support  versus  Investment 
Costs  using  Linear  Programming . 105 

Figure  23:  (L/D)n/(L/D)o  Value  Function . 157 

Figure  24:  Sample  Discrete  Evaluation  Measure  Calculation  Sheet . 183 

Figure  25:  Sample  Exponential  Evaluation  Measure  Calculation  Sheet . 184 

Figure  26:  Sample  Piecewise  Linear  Calculation  Sheet . 186 

Figure  27:  Sample  S-Curve  Calculation  Sheet . 186 

Figure  28:  Examples  of  S-Curve  Single  Dimension  Value  Functions  (Changes  in  Bias) 
. 196 

viii 


Figure  29:  Examples  of  S-Curve  Single  Dimension  Value  Functions  (Changes  in 

Steepness) . 196 

Figure  30:  Technological  Superiority/ Air  Force  Savings  Ratio . 199 

Figure  31:  Technological  Superiority/Be  Efficient/Dual  Use . 200 

Figure  32:  Technological  Superiority/Be  Efficient/Intermediate  Projects . 201 

Figure  33:  Technological  Superiority/Be  efficient/Multiuse . 202 

Figure  34:  Technological  Superiority/Be  Efficient/Uniqueness . 203 

Figure  35:  Technological  Superiority/Be  Efficient/User  Rating . 204 

Figure  36:  Technological  Superiority/Expand  Our  Frontier/Revolutionary . 205 

Figure  37:  Technological  Superiority/Expand  our  Frontier/New  Warfighting  Spectrum 

. 206 

Figure  38:  Reach/Gain/ Availability . 207 

Figure  39:  Reach/Gain/Missions  Enabled . 208 

Figure  40:  Reach/Gain/Payload-Range . 209 

Figure  41 :  Reach/Gain/Regeneration . 210 

Figure  42:  Reach/Gain/Response  Time  (Portion  of  Function  in  Figure  43) . 21 1 

Figure  43:  Reach/Gain/Response  Time  (Combined  Function  with  Figure  42) . 212 

Figure  44:  Power/Lethality/Gain/Effectiveness . 213 

Figure  45:  Power/Lethality/Gain/Survival . 214 

Figure  46:  Technological  Superiority/Programmatic  Risk . 215 

Figure  47:  Technological  Superiority/Simplify  Operations/Enables . 216 

Figure  48:  Technological  Superiority/Simplify  Operations/Interoperability . 217 

Figure  49:  Technological  Superiority/Simplify  Operations/Ratio  of  People . 218 

Figure  50:  Reach/Sustain/Achieve/Extend  Life . 219 

Figure  5 1 :  Reach/Sustain/ Availability . 220 

Figure  52:  Reach/Sustain/Missions  Enabled . 221 

Figure  53:  Reach/Sustain/Regeneration . 222 

Figure  54:  Reach/Sustain/Safety . 223 

Figure  55:  Reach/Sustain/Weight  Reduction . 224 

Figure  56:  Technological  Superiority/Win/Cost  to  Adversary . 225 

Figure  57:  Technological  Superiority/Win/Lead . 226 

Figure  58:  Technological  Superiority/Win/Warfare  Advantage . 227 

Figure  59:  Value  Contributions  for  “Support  the  Warfighter” . 242 


IX 


Figure  60:  Value  Contributions  to  “Reach” . 242 

Figure  61:  Value  Contributions  to  “Awareness” . 243 

Figure  62:  Value  Contributions  to  Power . 243 

Figure  63:  Value  Contributions  to  Technological  Superiority . 244 

Figure  64:  Value  Contributions  to  “Reach/Gain” . 244 

Figure  65:  Value  Contributions  to  “Reach/Sustain” . 245 

Figure  66:  Value  Contributions  of  “Be  Efficient”  under  Technological  Superiority . 245 

Figure  67:  Value  Contributions  of  “Expand  our  Frontier”  under  Technological 

Superiority . 246 

Figure  68:  Value  Contributions  of  “Simplify  Operations”  Under  Technological 

Superiority . , . 246 

Figure  69:  Value  Contributions  of  “Win”  Under  Technological  Superiority . 247 

Figure  70:  Sensitivity  of  Reach  (Left)  and  Awareness  (Right) . 249 

Figure  7 1 :  Sensitivity  of  Power  (Left)  and  Technological  Superiority  (Right) . 250 

Figure  72:  Sensitivity  of  Reach  Gain  (Left)  and  Reach  Sustain  (Right) . 251 

Figure  73:  Sensitivity  of  Afford  (Left)  and  Be  Efficient  (Right)  under  Technological 

Superiority . 252 

Figure  74:  Sensitivity  of  Expand  our  Frontier  (Left)  and  Programmatic  Risk  (Right) 

under  Technological  Superiority . 253 

Figure  75:  Sensitivity  of  Simplify  Operations  (Left)  and  Win  (Right)  under 

Technological  Superiority . 254 

Figure  76:  Sensitivity  of  Availability  (Left)  and  Missions  Enabled  (Right)  under  Reach 
under  Gain . e . 255 

Figure  77:  Sensitivity  of  Regeneration  (Left)  and  Response  Time  (Right)  under  Reach 
under  Gain . 256 

Figure  78:  Sensitivity  of  Payload -Range  under  Reach  under  Gain . 257 

Figure  79:  Sensitivity  of  Achieve/Extend  Life  (Left)  and  Availability  under  Reach  under 
Sustain . 258 

Figure  80:  Sensitivity  of  Missions  Enabled  (Left)  and  Regeneration  (Right)  under  Reach 
under  Sustain . 259 

Figure  8 1 :  Sensitivity  of  Safety  (Left)  and  Weight  Reduction  (Right)  under  Sustain  under 

Reach . 260 

Figure  82:  Sensitivity  of  Availability  (Left)  and  Missions  Enabled  (Right)  under 

Awareness  under  Gain . 261 

Figure  83:  Sensitivity  of  Regeneration  under  Awareness  under  Gain . 262 


x 


Figure  84:  Sensitivity  of  Effectiveness  (Right)  and  Survival  (Left)  under  Power  under 
Lethality . 263 

Figure  85:  Sensitivity  of  Dual  Use  (Left)  and  Intermediate  Products  (Right)  under 

Technological  Superiority  under  Be  Efficient . 264 

Figure  86:  Sensitivity  of  Multiuse  (Left)  and  Uniqueness  (Right)  under  Technological 
Superiority  under  Be  Efficient . 265 


Figure  87:  Sensitivity  of  User  Rating  under  Technological  Superiority  under  Be  Efficient 
. 266 

Figure  88:  Sensitivity  of  Revolutionary  (Left)  and  New  Warfighting  Spectrum  (Right) 
under  Technological  Superiority  under  Expand  our  Frontier . 267 

Figure  89:  Sensitivity  of  Enables  (Left)  and  Interoperability  (Right)  under  Technological 
Superiority  under  Simplify  Operations . 268 

Figure  90:  Sensitivity  of  Ratio  of  People  under  Technological  Superiority  under  Simplify 
Operations . 269 

Figure  91:  Sensitivity  of  Cost  to  Adversary  (Left)  and  Lead  (Right)  under  Technological 
Superiority  under  Win . 270 

Figure  92:  Sensitivity  of  Warfare  Advantage  under  Technological  Superiority  under  Win 
. 271 


xi 


List  of  Tables 


Table  1:  Aerospace  Dimensions  of  Research  and  Development . 17 

Table  2:  Summary  of  Scale  types  (Luce,  Bush  and  Galanter,  1963:  8-13) . 27 

Table  3:  Linear  Programming  Assumptions  (Winston,  1994:  53-54) . 35 

Table  4:  Model  Features  and  Ease  of  Use  Considerations  (Cetron,  Martino  and  Ropcke, 
1967:4-5) . 37 

Table  5:  Model  Evaluation  Criteria  (Bretschneider,  1993:  135;  Sounder,  1972:  B-528) .  38 

Table  6:  Weights  for  the  Air  Vehicles  Value  Model . 67 

Table  7:  Programs  Scored  Using  the  Air  Vehicles’  Value  Model . 71 

Table  8:  Dimensions  covered  by  Air  Vehicles  Value  Model . 76 

Table  9:  Characteristics  of  the  Air  Vehicles  Model  compared  to  R&D  Model  Features  in 
the  Literature . 78 

Table  10:  Overall  Scores  and  Relative  Rankings  for  Seven  Air  Vehicles’  Programs . 82 

Table  1 1 :  Approximate  Weight  changes  that  first  re-rank  Programs  (Note  -  Weights  do 
not  add  up  to  one  due  to  round  off) . 89 

Table  12:  Scored  Air  Vehicle  Program  Trends  under  Reach/Gain  Evaluation  Measures  93 

Table  13:  Change  in  Ranks  by  Changing  “Revolutionary”  for  Programs  Marked  with 

. 95 

Table  14:  Air  Vehicles’  Programs  Ranked  from  highest  to  lowest  Benefit  to  Cost  Ratio  96 


Table  15:  Change  in  Warfighter  Technological  Support  of  Air  Vehicles’  Programs 

Ranked  from  highest  to  lowest  Technological  Intensity . 98 

Table  16:  Technology  Leadership  Values  in  USAF  Strategic  Planning  Documents . 1 18 

Table  17:  Summary  of  Correlations  between  USAF  Strategic  Documents  and  Reach, 

Awareness,  Power,  and  Technological  Superiority . 119 

Table  18:  Detailed  Qualitative  Correlations  between  USAF  Strategic  Documents  and 

Reach,  Awareness,  Power,  and  Technological  Superiority . 120 

Table  19:  L/D  categories  for  Aerospace  Vehicles  (Source:  Personal  Interview  with  Mr. 
Russ  Osborne  in  Air  Vehicles  Directorate  of  AFRL) . 156 


xii 


Abstract 


A  technology  selection  model  was  developed  for  the  Air  Vehicles  Directorate  of 
Air  Force  Research  Laboratory.  The  model  was  developed  and  demonstrated  to  aid  in 
addressing  the  question  of  what  technologies  Air  Vehicles’  should  invest  in  to  remain 
consistent  with  Air  Force  values.  Both  Value-Focused  Thinking  and  optimization 
approaches  were  used  to  identify  the  value  of  Air  Vehicles’  technology,  to  provide 
insights  to  Air  Vehicles’  decision-makers,  to  determine  where  value  gaps  might  exist 
with  the  scored  alternatives,  and  to  determine  how  sensitive  the  model  was  to  changes. 
As  a  demonstration  of  the  approach,  seven  technologies  were  scored,  representing  all 
three  of  the  Air  Vehicles’  Integrated  Concepts.  In  the  process,  the  support  to  the 
warfighter  due  to  technological  change  was  quantified  and  analyzed. 


TECHNOLOGY  SELECTION  FOR  THE  AIR  FORCE  RESEARCH 
LABORATORY  AIR  VEHICLES  DIRECTORATE:  AN  ANALYSIS  USING 

VALUE  FOCUSED  THINKING 

Chapter  1 :  Introduction 

Background 

Organizations  that  perform  research  and  development  (R&D)  are  required  to 
justify  their  investments.  In  this  era  of  decreasing  government  budgets,  increasing  global 
competition  and  decreasing  technological  life,  proper  selection  of  R&D  projects  is  ever 
more  critical.  Decision  makers  must  show  R&D  adds  value  in  support  of  a  larger  mission 
(or  show  an  improvement  of  shareholder  wealth  in  the  case  of  private  industry).  This  is 
particularly  true  for  the  public  sector  organizations,  such  as  the  Department  of  Defense 
(DoD).  Failure  to  show  value  added  can  result  in  funding  cuts  and  the  potential 
dismantling  of  the  R&D  organization.  In  the  short  term,  an  organization’s  technological 
position  may  not  suffer.  In  the  long  term,  however,  failure  to  perform  R&D  will  result  in 
obsolescence  of  weapons  in  the  DoD.  In  the  private  sector,  poor  R&D  can  prevent 
industry  from  competing  successfully  with  other  commercial  firms.  This  thesis  proposes 
and  demonstrates  a  methodology  for  the  Air  Vehicles  Directorate  (VA)  of  the  Air  Force 
Research  Laboratory  (AFRL)  to  construct  the  most  valuable  portfolio  of  programs  to 
meet  future  United  States  Air  Force  (USAF)  warfighting  requirements.  Further,  it 
establishes  a  value  measure  for  the  proposed  VA  current  and  future  projects. 

In  Aug  98,  the  Air  Vehicles  Directorate  requested  the  Air  Force  Institute  of 
Technology’s  (AFIT)  help  to  analyze  the  following  questions: 


“What  direction  in  technology  R&D  should  the  Air  Vehicle  Directorate  take  to 
be  most  consistent  with  Air  Force  values?” 

The  question  of  what  to  invest  in  is  not  new.  For  example,  the  Flight  Dynamics 

Laboratory  (a  precursor  to  the  Air  Vehicles  Directorate)  was  posing  a  similar  question 

about  aeronautics  programs  over  30  years  ago  (Nutt,  A.B.  1965:103-1 12).  The  question 

being  asked  now  is  analogous  to  the  questions  asked  then.  However,  today’s  emphasis 

on  “aerospace”  rather  than  just  aeronautics  implies  both  aeronautical  projects  and  space 

projects. 

Determining  which  technologies  to  pursue  given  a  large  number  of  possible 
investments  can  be  both  difficult  and  bewildering  without  a  systematic  process  of 
evaluation.  If  it  were  possible  to  know  the  future  with  certainty,  it  would  be  a  much  more 
straightforward  matter  to  structure  technology  programs  to  meet  DoD  mission  needs. 
However,  not  only  is  there  uncertainty  about  future  needs,  there  is  also  uncertainty  about 
whether  a  particular  technology  can  and  will  even  be  developed. 

In  the  DoD,  deciding  what  is  valuable  about  a  particular  R&D  program  is  more 
difficult  than  in  industry.  In  the  commercial  world,  R&D  is  generally  performed  with  a 
profit  motive  in  mind,  which  at  least  can  be  estimated  via  market  research  concepts  and 
analysis.  In  the  DoD,  a  valuable  weapon  is  one  that  helps  to  win  or  to  deter  war.  In  the 
DoD,  R&D  supports  current  and  future  warfighting  missions,  a  difficult  element  to 
predict.  The  consequences  of  poor  R&D  can  be  much  higher  than  the  failure  of  a 
business.  As  devastating  as  the  loss  of  a  business  might  be,  the  loss  of  national 
sovereignty  is  much  more  serious.  The  most  effective  measure  of  a  successful  R&D 
program  for  a  nation  takes  place  after  the  technology  is  tested  in  battle,  where  loss  of  life, 
territory,  or  national  sovereignty  are  the  possible  price  of  poor  decision  making. 
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In  Technology  and  War,  van  Creveld’s  states  “that  war  is  completely  permeated 

by  technology  and  governed  by  it”  (van  Creveld,  Martin  1989:  1).  A  second  rate  R&D 

program  is  measured  directly  in  lives  lost  and,  ultimately,  in  the  rise  and  fall  of  nations. 

The  introduction  of  1997  Defense  Technology  Area  Plan  states  the  United  States  cannot 

afford  to  find  out  its  technology  is  second  best  by  losing  a  military  engagement: 

Technological  superiority  has  been,  and  continues  to  be,  a  cornerstone  of 
our  national  military  strategy.  Technologies  such  as  radar,  jet  engines, 
nuclear  weapons,  night  vision,  smart  weapons,  stealth,  the  Global 
Positioning  System,  and  vastly  more  capable  information  management 
systems  have  changed  warfare  dramatically.  Today's  technological  edge 
allows  us  to  prevail  across  the  broad  spectrum  of  conflict  decisively  and 
with  relatively  low  casualties.  Maintaining  this  technological  edge  has 
become  even  more  important  as  the  size  of  U.S.  forces  decreases  and  high- 
technology  weapons  are  now  readily  available  on  the  world  market.  In  this 
new  environment,  it  is  imperative  that  U.S.  forces  possess  technological 
superiority  to  achieve  and  maintain  the  dominance  displayed  in  Operation 
Desert  Storm.  The  technological  advantage  we  enjoy  today  is  a  legacy  of 
decades  of  investment  in  science  and  technology  (S&T).  Likewise,  our 
future  warfighting  capabilities  will  be  substantially  determined  by  today's 
investment  in  S&T  (The  Military  Critical  Technologies  List.  1997 
Defense  Technology  Area  Plan,  1997:  Introduction). 

Predicting  effectiveness  for  a  technology,  which  has  not  yet  been  invented,  against  an 

unknown  enemy,  for  an  unknown  battle,  in  an  unknown  environment  is  a  daunting  task. 

Yet,  it  is  necessary  to  plan  for  an  uncertain  future. 

Purpose 

R&D  investment  in  the  DoD  must  effectively  provide  the  warfighter  the  right 
technology.  The  AFRL,  with  its  ten  Directorates,  is  the  organization  that  oversees  and 
performs  R&D  for  the  United  States  Air  Force  warfighters.  The  purpose  of  this  thesis  is 
to  develop  a  method  to  analyze  future  technology  selections  for  the  Air  Vehicles 
Directorate.  Further,  the  analysis  method  must  identify  technology  alternatives  that  are 
consistent  with  choices  the  VA  decision-makers  believe  are  critical  “to  keep  our  Air 
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Force  the  best  in  the  world”  (. AFRL  Mission,  1998).  It  must  be  possible  to  identify 
technology  alternatives  in  a  documented  and  traceable  fashion  and  propose  a  means  to 
overcome  shortfalls  in  the  decision-maker’s  values  and  objectives.  The  technology 
programs  proposed  must  also  be  linked  to  investments  to  provide  VA  the  means  to 
implement  the  decision-maker’s  values. 

Scope 

The  Air  Vehicles’  Director  determined  the  study  should  start  with  1999  and  look 
10  years  into  the  future.  Various  technology  programs  make  up  the  possible  alternatives 
VA  may  choose  to  support  in  a  limited  funding  environment.  The  ultimate  goal  of  this 
research  effort  is  to  help  all  ten  directorates  of  the  AFRL.  In  view  of  the  available 
resources  to  conduct  the  study,  however,  the  Air  Vehicles  analysis  was  performed  to 
serve  as  a  framework  for  a  larger  AFRL  future  effort.  At  the  same  time,  it  will  help  Air 
Vehicles  to  select  the  best  technologies  for  their  Directorate. 

The  study  was  limited  to  deterministic  evaluation  of  the  programs.  It  is 
understood  that  predictions  about  technology  will  be  uncertain.  One  reason  for  the 
limitation  was  consideration  of  available  resources  for  this  effort.  The  second  reason  was 
that  it  is  important  to  first  determine  what  the  critical  uncertainties  are  before 
implementing  uncertainty  techniques  in  the  model.  If  uncertainty  is  too  small,  it  may 
have  no  effect  on  the  modeling  results  and  might  be  wasting  the  valuable  time  of 
decision-makers  and  other  technical  experts  in  Air  Vehicles  during  elicitation  of 
information.  Clearly,  only  those  variables  that  have  large  uncertainty  (as  identified  by 
sensitivity  analysis)  should  be  considered  for  analysis  under  uncertainty. 
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Values  (what  is  important  to  VA  about  choosing  a  technology  program)  are  used 
in  a  value  model  to  quantify  how  well  the  alternatives  support  the  Air  Force  warfighter. 
The  result  of  quantifying  programs  allows  VA  to  identify  the  “best”  programs  for 
investment.  It  also  allows  “what  if’  testing  or  determination  of  how  robust  a  particular 
decision  policy  is  using  sensitivity  analysis.  Finally,  shortfalls  in  the  overall  VA 
portfolio  (defined  as  low  scoring  alternatives  or  values  with  no  alternatives  to  satisfy 
them)  can  be  uncovered  and  dealt  with.  These  insights  will  be  provided  to  the  Air 
Vehicle’s  decision-makers  to  aid  them  in  selecting  technology  program  investments. 
Summary  and  Organization 

Chapter  2  documents  theoretical  considerations  from  the  literature  for  the  project. 
It  begins  by  identifying  the  dimensions  of  R&D  and  those  peculiar  to  the  military  and  Air 
Vehicles.  Next,  technology  selection  models  are  discussed  and  some  examples  of 
multidimensional  models  are  discussed.  Afterward,  in-depth  discussion  is  provided  on 
value  focused  thinking  and  resource  allocation  as  it  applies  to  R&D.  The  chapter  wraps 
up  with  a  discussion  of  the  characteristics  of  past  R&D  models  and  identifies  limitations 
of  these  models. 

Chapter  3  applies  the  results  found  in  the  Chapter  2  literature  review.  First,  an 
overview  of  development  of  a  model  capable  of  selecting  technology  for  Air  Vehicles  is 
presented.  Next,  detail  discussion  is  provided  on  developing  a  value  hierarchy  and  a 
value  model.  In  the  process,  some  issues  peculiar  to  modeling  R&D  are  discussed.  With 
the  value  model  completed,  assumptions  for  the  model  are  explained.  Then,  using  the 
value  model  to  score  alternatives  is  explained.  The  chapter  wraps  up  by  discussing 
methods  of  evaluating  the  modeling  results. 
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Chapter  4  shows  results  of  an  illustrative  analysis  on  a  subset  of  Air  Vehicles’ 
programs  while  Chapter  5  concludes  the  effort.  Several  appendices  provide  more 
detailed  information  of  key  areas  in  the  thesis  and  are  cited  in  appropriate  places 
throughout  this  document. 
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Chapter  2:  Literature  Review 


Introduction 

Biondi  and  Galli  argue  that  innovation  in  technology  is  caused  by  the  efforts  of 
people.  If  innovation  has  no  causality,  there  is  no  basis  for  technology  forecasting. 
(Biondi  and  Galli,  1992:  580).  In  this  chapter,  the  literature  is  reviewed  to  determine 
what  dimensions  make  up  aerospace  research  and  development.  The  literature  is  first 
used  to  define  science  and  technology.  It  is  used  to  identify  basic  research,  applied 
research  and  development  research.  Next,  dimensions  for  basic  research, 
applied/development  research,  and  all  three  types  of  research  are  identified.  A  model  for 
technological  change  is  discussed  and  then  technological  dimensions  specific  to 
aeronautics,  astronautics  and  the  military  are  discussed. 

Having  discussed  dimensions  of  aerospace  R&D,  technology  selection  models  are 
reviewed  and  some  examples  of  multidimensional  scoring  studies  are  provided.  The 
discussion  is  followed  by  an  in-depth  review  of  Value-Focused  Thinking  and  resource 
allocation  models.  The  review  is  completed  by  examining  characteristics  and  limitations 
of  R&D  technology  selection  models. 

Aerospace  Research  and  Development  (R&D) 

Definitions 

Science  can  be  defined  as  acquiring  of  “  . . .  general,  fundamental  and  abstract 
forms  of  knowledge”  (Organization  for  Economic  Co-operation  and  Development,  1992: 
16).  The  objective  of  basic  scientific  research,  is  to  gain  knowledge  that  is  more 
comprehensive  or  to  gain  understanding  of  the  subject  under  study,  without  specific 
applications  in  mind.  In  industry,  basic  research  is  defined  as  research  that  advances 
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scientific  knowledge  but  does  not  have  specific,  immediate  commercial  objectives, 
although  it  may  be  in  fields  of  present  or  potential  commercial  interest  (National  Science 
Foundation,  Science  &  Engineering  Indicators  - 1996,  NSF  96-21,  1996). 

Basic  research  is  rarely  valuable  by  itself,  but  requires  linkage  to  developments  in 
technology  (Pavitt,  1991:  1 12).  According  to  van  Creveld,  “technology  is  perhaps  best 
understood  as  an  abstract  system  of  knowledge,  an  attitude  towards  life  and  a  method  for 
solving  its  problems”  (van  Creveld,  1989:  312).  The  Organization  for  Economic  Co¬ 
operation  and  Development  observes  technology  is  " . . .  specific  and  practical,”  and 
“requires  the  development  of  a  number  of  ‘transfer  sciences’  situated  at  the  interface 
between  basic  knowledge  and  the  solution  of  concrete  problems  arising  from  economic 
and  social  needs"  (Organization  for  Economic  Co-operation  and  Development,  1992:  16). 
A  link  exists  between  science  and  technology  in  that  technology  builds  on  what  science 
has  discovered  and  it  applies  advances  in  science  to  solve  human  problems. 

The  National  Science  Foundation  defines  two  types  of  technology  development. 
Applied  research  is  aimed  at  gaining  knowledge  or  understanding  to  determine  the  means 
by  which  a  specific,  recognized  need  may  be  met.  In  industry,  applied  research  includes 
investigations  oriented  toward  discovering  new  scientific  knowledge  that  has  specific 
commercial  objectives  with  respect  to  products,  processes,  or  services.  Development 
research  is  the  systematic  use  of  the  knowledge  or  understanding  gained  from  research 
directed  toward  the  production  of  useful  materials,  devices,  systems,  or  methods, 
including  the  design  and  development  of  prototypes  and  processes  (National  Science 
Foundation,  Science  &  Engineering  Indicators  - 1998,  NSF  96-21,  1998).  A  link  can  be 
seen  between  applied  research  and  development  research  in  that  the  first  feeds  the  latter. 


8 


Dimensions  Specific  to  Basic  Research 

Martin  finds  basic  research  contributes  to  many  dimensions  of  human  activity  and 
knowledge.  It  can  contribute  to  the  scientific  stock  of  knowledge,  it  can  be  used  to  train 
and  improve  skills  of  people,  it  can  improve  technological  capabilities  and  it  can 
contribute  to  the  culture  of  society.  Scientific  contributions  are  not  limited  to  just  the 
field  they  originated  in,  but  can  affect  many  fields  of  study.  Further,  the  work  can  be 
theoretical,  empirical  or  methodological.  The  advances  made  by  research  can  be 
revolutionary  or  incremental.  Education  to  people  includes  the  corporate  knowledge  and 
the  ability  to  solve  complex  problems.  Technological  capabilities  can  include  new 
products  and  services,  new  instrumentation,  and  new  methodologies  that  can  be  used  in 
applied  research  efforts.  The  results  of  basic  research  can  result  in  direct  economic  and 
social  benefits  (Martin,  1996:  346). 

Evolutionary  change  takes  place  in  step-by-step  increments,  with  each  increment 
building  on  the  knowledge  of  the  previous  increment,  resulting  in  cumulative  learning. 
“Technological  discontinuity”  is  possible,  though  rare  (Biondi  and  Galli,  1992:  583).  It  is 
defined  as  “radical  innovation,  which  replaces  a  wide  range  of  earlier  techniques, . . .  and 
leads  to  revolutionary  changes  in  a  system’s  economic  and  institutional  arrangements” 
(Biondi  and  Galli,  1992:  585). 
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Another  characteristic  of  basic  research  is  risk. 


When  you  are  organized  to  apply  knowledge,  you  require  a  high  degree  of 
certainty  from  the  outset.  In  basic  research,  everything  is  just  the 
opposite.  What  you  need  at  the  outset  is  a  high  degree  of  uncertainty; 
otherwise  it  is  not  likely  to  be  an  important  problem  (National  Academy  of 
Sciences,  1982:  30). 

Since  R&D  projects  do  not  always  succeed,  experts  typically  estimate  (often 
subjectively)  probability  of  success.  The  probability  of  success  is  subjectively 
determined  by  experts  (Bretschneider,  1993:  129). 

Dimensions  Specific  to  Applied  and  Development  Research 

Attributes  that  characterize  a  device  developed  by  an  R&D  effort  fall  into  two 
general  groups  (although  they  have  different  names  in  the  literature).  Alexander  and 
Nelson  divide  technology  characteristics  between  performance  parameters  that  provide 
value  to  the  user  and  technical  parameters  that  “make  the  performance  parameters 
possible  .  . .”  (Alexander  and  Nelson,  1973:  190).  Alexander  and  Mitchel  divide 
technology  characteristics  between  product  user  and  product  producer  (Alexander  and 
Mitchel,  1985:  162).  Majer  categorizes  attributes  of  technologies  into  those  that  describe 
“its  condition  or  nature  and  that  reflect  its  use  or  purpose”  (Majer,  1985:  337). 

Biondi  and  Galli  identify  six  technological  trajectories  they  maintain  are 
historically  stable:  1)  cost  reduction,  2)  incidence  of  capital  cost,  3)  space  time  dimension 
of  goods  and  services,  4)  incidence  of  specific  resource  consumption,  5)  lifetime,  and  6) 
size/scale  effects  (Biondi  and  Galli,  1992:  585-587). 

Cost  reduction,  due  to  "innovation  selection,"  results  in  more  costly  processes 
being  rejected  in  favor  of  less  costly  processes.  It  is  possible  for  cost  reduction  to  stop  at 
some  point,  but  it  cannot  be  reversed.  Capital  costs  are  investments  in  artificial  tools 
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used  to  reduce  labor  and  resources  in  a  process.  Some  investments  in  R&D  have  the  goal 
of  reducing  this  cost. 

Time  space  dimensions  in  goods  and  services  are  reductions  in  volume,  weight,  or 
time  to  exploit  services.  A  reduction  in  volume  or  weight  takes  place  when  a  technology 
can  perform  the  same  job  in  less  volume  or  with  less  weight  than  a  previous  technology. 
Reductions  in  time  takes  place  using  special  equipment,  which  automates  human 
activities. 

Incidence  of  Specific  Resource  consumption  is  defined  as  a  reduction  in  resource 
inputs  while  maintaining  quality  and  quantity  of  a  process.  Lifetime  is  defined  as  a 
process  with  longer  useable  time  a  technology  can  be  use  for.  A  product  with  a  longer 
lifetime  is  preferred  to  one  with  less  lifetime.  All  processes  deteriorate  over  time  and 
eventually  become  too  costly  to  maintain.  A  process  can  be  increased  in  size,  sometimes 
resulting  in  economies  of  scale.  All  size  and  scaling  effects  must  result  in  reductions  in 
total  cost  to  be  useful. 

Dimensions  General  to  Basic,  Applied  and  Development  Research 

The  trajectories  identified  by  Biondi  and  Galli  of  “preferential  paths”  followed  by 
technology,  can  be  both  qualified  and  quantified.  Quality  is  defined  as  “a  characteristic 
set  of  performances  appreciated  by  users.”  The  authors  imply  quantity  is  a  specific 
measurement  level  of  a  quality.  Examples  of  qualities  of  a  vehicle  are  roominess,  image, 
speed,  fuel  consumption  and  price.  The  quantity  of  a  specific  quality  can  be  measured  as 
a  continuous  set  of  values  (such  as  price  or  speed)  or  it  can  be  measured  as  discrete 
discontinuous  levels.  An  example  of  a  discrete  measurement  of  a  television  is  black  and 
white  or  color  (Biondi  and  Galli,  1992:  582). 


11 


Werner  and  Sounder  further  breakdown  quantitative  and  qualitative  R&D 
effectiveness  measures.  Measures  can  be  based  on  either  subjective  judgements  or 
objective  data.  They  categorize  metrics  as  either  quantitative-objective,  quantitative- 
subjective  and  qualitative-subjective.  The  qualitative-objective  category  is  ignored  since 
the  authors  consider  it  to  be  a  contradiction  in  terms  (Werner  and  Sounder,  1997:  34). 

Quantitative-objective  metrics  are  "numerical  indicators  of  R&D  performance 
using  well-defined  algorithms  that  focus  on  tangible,  countable  dimensions.” 
Quantitative-subjective  metrics  are  "based  on  intuitive  judgments  that  are  converted  to 
numbers.”  Qualitative-subjective  measures  do  not  attempt  to  mathematically  compare 
different  levels  of  the  measure.  Raw  evaluations  of  expert  assessors  are  relied  upon  and 
no  attempt  is  made  to  convert  the  judgements  to  numbers.  (Wemer  and  Sounder,  1997: 
34-37). 

Technological  Change 

Alexander  and  Mitchel  model  technological  change  from  the  views  of  the  producer  and 
the  user.  They  develop  two  systems  of  relationships,  which  transform  inputs  to  outputs 
as: 


F(RP,P,t)=  0 
G(Ru,U,t)  =  0 


Equation  1 


where  P  and  U  are  vectors  of  “product”  and  “use  variables;  Rp  a  vector  of 
factors  inputs  (resources)  of  the  product  producer,  and  Ru  a  vector  of 
resources  consumed  by  the  product  user  to  produce  the  vector  of  outputs 
U.  The  vector  Ru  includes  as  an  input  the  output  of  the  product  producer 
as  well  as  additional  inputs  such  as  labor,  materials,  and  energy 
(Alexander  and  Mitchell,  1985:  167). 

The  authors  then  state: 
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Technological  change  will  then  be  -dRp /dt  ...  and  -  dRv  /dt  . . .,  with 
P  and  U  constant.  Alternatively,  technological  change  can  be  defined  as 
dP/dt  and  dU /dt  ...  with  the  respective  R  vectors  constant  (Alexander 
and  Mitchell,  1985:  167). 

The  authors  analyze  milling  machines  (data  spanning  over  100  years,  although 
with  some  gaps),  commercial  turbine-powered  transports,  and  turbine  engines.  In  their 
transport  data,  the  vector  P  is  made  up  of  elements  such  as  “lift-to-drag  ratio,  structural 
efficiency,  engine  thrust,  wing  loading,  and  energy  content  of  fuel,”  (Alexander  and 
Mitchell,  1985:  167).  The  U  vector  are  quantities  such  as  “seat-miles  or  ton-miles,  noise, 
pollution,  and  the  costs  of  obtaining  (or  avoiding)  these  activities”  (Alexander  and 
Mitchell,  1985:  168).  The  authors  note  that  the  use  of  technical  parameters  as  a  proxy  to 
measure  value  and  performance  to  the  user  do  not  properly  measure  technological  change 
from  the  user’s  point  of  view. 

Dimensions  Specific  to  U.S.  Air  Force  and  Aerospace  R&D 

The  Air  Vehicles’  strategic  plan,  Air  Force  Research  Laboratory  Air  Vehicles 
Directorate  2010:  Strategic  Business  Plan  ( Working  Draft  #3)  is  critical  to  understanding 
what  is  important  to  Air  Vehicles  in  R&D.  This  document  identifies  Air  Vehicles’  three 
Integrated  Concepts  of  Aircraft  Sustainment,  Space  Superiority,  and  Uninhabited  Air 
Vehicles  (UAV)  (AFRL/VA  Corporate  Planning  Group,  1998:  15-17).  It  also  brings  out 
Air  Vehicle’s  commitment  to  maintain  competency  in  aerospace  technologies  (AFRL/VA 
Corporate  Planning  Group,  1998:  18).  The  “Strategic  Position”  states  Air  Vehicles  will 
“deliver  the  best  air  vehicle  technologies  for  aerospace  dominance  against  all  threats  . . .” 
(AFRL/VA  Corporate  Planning  Group,  1998:  5)  or  support  the  warfighter  with 
technology.  The  document  highlights  the  importance  of  preeminence  in  research 
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performed  by  talented  and  dedicated  people  (AFRL/VA  Corporate  Planning  Group, 

1998:  6). 

Other  information  from  Air  Vehicles  is  the  programs  that  are  proposed  and 
currently  in  work.  Documentation  of  programs  can  be  found  in  the  Air  Vehicles 
Strategic  Plan  (AFRL/VA  Corporate  Planning  Group,  1998:  15-17)  and  it  can  be  found  in 
FY98  Air  Vehicles  Technology  Area  Plan  Annexes  (FY98  Air  Vehicles  Technology  Area 
Plan  Annexes,  1998:  Annex  B  and  C).  The  most  current  information  is  available  from 
the  Air  Vehicles’  Plans  and  Programs  office,  which  consists  of  the  latest  briefings  and 
individual  “Internal  Program  Summary”  sheets.  Additionally,  the  most  current  programs 
are  listed  in  the  Air  Vehicles’  roadmap  briefings  (Boudreau,  February  1999;  Weber, 

1999;  Ziegler,  January  1999). 

Air  Force  Basic  Doctrine  contains  the  classic  principles  of  war  with  definitions. 
Because  the  ultimate  goal  of  Air  Vehicles  is  to  support  the  warfighter,  the  principles  of 
war  provide  an  excellent  framework  for  thinking  about  how  the  warfighter  might  employ 
technology  to  win  a  war.  The  classic  principles  of  war  are  unity  of  command,  objective, 
offensive,  mass,  maneuver,  economy  of  force,  security,  surprise,  and  simplicity  (Air 
Force  Basic  Doctrine,  Sep  97:  12-21). 

Joint  Vision  2010  identifies  four  operational  concepts  of  dominant  maneuver, 
precision  engagement,  full  dimensional  protection,  and  focused  logistics  as  challenges  for 
all  services  of  the  military  in  the  future.  At  the  same  time,  it  documents  the  importance 
of  technological  superiority,  how  it  is  achieved  and  why  it  is  important.  Global 
Engagement  identifies  and  defines  the  USAF’s  core  competencies  that  work  closely  with 
Joint  Vision  2010.  The  core  competencies  are  rapid  global  mobility,  precision 


14 


engagement,  global  attack,  air  and  space  superiority,  information  superiority,  and  agile 
combat  support. 

Another  group  of  documents,  which  many  of  the  USAF  documents  build  on,  is 
New  World  Vistas.  This  document  brings  out  the  concepts  of  global  awareness,  dynamic 
planning  and  execution  control,  mobility,  projection  of  lethal  and  sublethal  power,  space 
operations  and  people  as  important  tenants  for  winning  future  wars.  These  documents  are 
future  thinking,  strategic  planning  documents  and  implementation  plans.  They  provide 
important  clues  of  what  is  important  to  the  USAF  and  Air  Vehicles. 

The  Air  Force  Strategic  Plan  contains  the  mission  of  the  USAF  and  the  vision 
statement.  The  vision  of  the  USAF  is  defined  as  “Air  Force  people  building  the  world’s 
most  respected  Air  and  Space  Force  global  power  and  reach  for  America.”  Global  power 
and  reach  are  key  concepts  of  what  is  valuable  to  the  USAF.  Reach  and  power  are 
fundamental  properties  air  forces  bring  to  a  battlefield. 

Another  interesting  study,  which  examines  the  past  to  understand  the  future,  was 
written  by  van  Creveld.  The  author  examines  the  impact  of  technology  on  warfare  from 
the  beginning  of  recorded  human  history  to  the  present.  Understanding  how  technology 
has  affected  warfare  over  much  of  recorded  human  history  provides  indications  of  how 
technology  may  affect  future  warfare. 

Other  specifics  to  aeronautical  R&D  are  standard  texts  on  aeronautics  used  to 
understand  standard  aeronautical  design  parameters  such  as  the  lift  to  drag  ratio 
(Anderson,  1985:  256)  or  range  of  an  aircraft  (Anderson,  1985:  298-302).  Additionally, 
Air  Vehicles  is  researching  space  capable  vehicles  (AFRL/VA  Corporate  Planning 
Group,  1998:  15-17),  which  is  part  of  astronautics.  As  such,  launch  vehicle  design  is 
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becoming  more  important.  Launch  vehicle  design  basics  and  considerations  are  well 
known  (see  Ryan  and  Townsend,  1997:  192-198  for  review  and  lessons  learned  from 
some  launch  vehicles).  Current  launch  vehicle  performance  (both  expendable  launch 
vehicles  and  the  space  shuttle)  can  be  found  on  the  internet  (NASA:  Kennedy  Space 
Center,  1999). 

Another  standard  work  specific  to  military  aerospace  systems  discusses  concepts 
of  how  survivable  an  aircraft  is  to  battle  damage  (Ball,  1985,  1-2).  The  probability  of  an 
aircraft  surviving  a  combat  sortie  is  a  function  of  its  susceptibility  and  its  vulnerability. 
Susceptibility  is  defined  as  the  probability  the  aircraft  is  hit;  it  covers  detection  and  the 
ability  of  an  adversary  to  hit  the  aircraft  with  a  weapon  system.  Vulnerability  is  the 
probability  the  aircraft  is  killed  given  that  it  has  been  hit.  This  concerns  how  much  battle 
damage  is  caused  by  an  adversary’s  weapon  and  whether  or  not  a  vital  system  of  the 
aircraft  has  been  damaged. 

Summary 

Table  1  summarizes  the  many  dimensions  that  apply  to  aerospace  R&D.  The 
dimensions  can  be  broken  down  into  specific  to  Basic  research,  specific  to  Applied  and 
Development  research,  specific  to  all  of  R&D  or  specific  to  a  field  of  study.  For  Air 
Vehicles,  field  specific  technology  covers  the  military,  aeronautics  and  astronautics. 
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R&D  Dimension 

Specific  to 

Authorise 

Aerospace  Specific  Measurements 

Aeronautics  and 
Astronautics 

Anderson;  Ryan  and 
Townsend;  Ball 

Contribute  to  Society 

Basic  Research 

Martin 

Evolutionary  versus  Revolutionary 

Basic  Research 

Martin;  Biondi  and  Galli 

Improve  Capability 
-Theoretical 

-  Empirical 

-  Methodological 

Basic  Research 

Martin 

Performance  versus  Technical 

Applied  and 

Development 

Research 

Alexander  and  Nelson; 
Alexander  and  Mitchel; 
Majer 

Programs 

All  Research 

Air  Vehicles’  Documents 

Qualitative  and  Quantitative 

-  Quantitative-objective 

-  Quantitative-subjective 

-  Qualitative-subjective 

Basic,  Applied 
and  Development 
Research 

Biondi  and  Galli; 

Werner  and  Sounder 

Risk 

Basic  Research 

National  Academy  of 
Science;  Bretschneider 

Scientific  Knowledge 

Basic  Research 

Martin 

Technological  Change 

Basic,  Applied 
and  Development 
Research 

Alexander  and  Mitchel 

Technological  Trajectories 

-  Cost  Reduction 

-  Incidence  of  Capitol  Cost 

-  Space  Time  Dimension  of  goods 
and  services 

-  Incidence  of  Specific 

Resource  Consumption 

-  Lifetime 

-  Size/scale  effects 

Applied  and 

Development 

Research 

Biondi  and  Galli 

Train  and  Educate 

Basic  Research 

Martin 

Military  Specific 

-  U.S.  Air  Force  Strategic  Planning 

-  U.S.  Air  Force  Doctrine 

-  Historical  Analysis  of  War 

T-LI*  -f  _  A _ 

Military  Field 

AFRL  Air  Vehicles  2010: 
Strategic  Business  Plan 
(Working  Draft  #3);  Air 

Force  Basic  Doctrine;  Joint 
Vision  2010;  Global 
Engagement;  New  World 
Vistas;  Air  Force  Strategic 
Plan;  van  Creveld 

Table  1:  Aerospace  Dimensions  of  Research  and  Development 
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Technology  Selection  Models 

Having  identified  the  dimensions  of  aerospace  R&D,  a  model  is  needed  to  support 
technology  selection.  Bretschneider  defines  two  types  of  analysis  in  R&D  assessments. 
He  defines  “ex  ante”  R&D  as  when  “  . . .  evaluation  occurs  prior  to  initiation  of  the 
research  or  development  activity”  and  “ex  post”  R&D  as  when  “  . . .  evaluation  occurs 
after  a  project  has  been  completed  and  therefore  focuses  on  outcomes  and  impacts.”  The 
principle  use  of  ex  ante  analysis  is  to  select  projects  (Bretschneider,  1993:  123).  Since 
this  thesis  is  an  ex  ante  analysis  for  Air  Vehicles,  discussion  will  be  confined  to  ex  ante 
analysis. 

Ex  ante  studies  can  be  divided  into  valuation  or  benefit  measurement  models  and 
resource  allocation  models  (Bretschneider,  1993,  124;  Baker  and  Freeland,  1975,  1 164). 
Valuation  models  are:  1)  models  that  develop  a  measure  of  value  through  a  comparative 
technique,  2)  models  based  on  obtaining  a  multidimensional  score,  or  3)  techniques  that 
link  a  project’s  value  to  the  overall  economic  objective  of  the  firm  or  organization. 
Resource  allocation  models  are:  1)  constrained  optimization  models,  2)  emulations  of 
organizational  and  human  processes  (simulation),  or  3)  Ad-hoc  in  nature  (Bretschneider, 
1993:  124). 

Comparative  approaches  examine  two  measures  at  a  time  and  are  time  consuming 
if  many  projects  are  to  be  compared.  If  comparisons  are  binary,  then  for  n  projects, 
n(n- 1)  /  2  comparisons  are  required.  The  resulting  relative  results  can  be  difficult  or 
impossible  to  aggregate  for  analysis.  Computer  programs  are  used  to  check  for 
consistency  between  evaluators,  but  it  is  often  difficult  to  maintain  consistency  because 
there  is  no  standard  on  which  to  base  an  evaluation  (Bretschneider,  1993:  127-128). 
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In  Bretschneider’s  approach,  multidimensional  scoring  models  develop  a  small 
number  of  criteria  for  R&D  projects.  Each  project  is  then  scored  by  experts  for  each 
attribute.  A  formula  is  developed  to  aggregate  the  attributes  across  multiple  dimensions 
(Bretschneider,  1993:  128). 

Value  focused  thinking  (VFT)  and  the  analytical  hierarchy  process  (AHP)  are  two 
types  of  multidimensional  scoring  models.  VFT  is  described  by  Keeney  and  by 
Kirkwood  (Keeney,  1992;  Kirkwood,  1997).  VFT  places  values  in  a  hierarchical 
structure  and  quantifies  the  values  with  evaluation  measures,  known  as  a  value  model. 
Alternatives  are  scored  using  the  value  model  allowing  quantification  of  how  well  the 
values  are  being  achieved.  The  AHP  (Saaty,  1980)  structures  priorities,  which  are  similar 
to  values  in  a  hierarchy.  Pairwise  comparisons  of  the  alternatives  are  combined  with  the 
hierarchy  to  develop  a  ranking  of  the  alternatives. 

Proponents  for  VFT  and  AHP  are  documented  in  the  literature.  Bard  concludes 
that  AHP  is  simpler  to  use  for  a  decision-maker  inexperienced  in  either  VFT  or  AHP  and 
that  AHP  is  better  for  a  large  number  of  attributes  that  can  only  be  measured  on  a 
subjective  scale  (Bard,  1992:  120).  A  weakness  of  VFT  is  the  inability  of  it  to 
systematically  check  for  consistency  of  judgements  (Belton,  1986:  18).  However,  AHP 
is  known  to  have  theoretical  problems,  which  makes  it  less  attractive  for  use  than  VFT. 
Belton  compares  VFT  with  the  AHP  and  finds  the  AHP  has  major  weaknesses  in  the  way 
questions  are  asked  to  determine  criteria  weights  and  in  the  assumption  that  the 
measurement  scores  can  be  compared  using  a  ratio.  Further,  the  AHP  always  results  in 
additive  weighted  value  functions  and  should  not  be  used  for  risky  decisions,  where  as 
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VFT  is  not  restricted  in  these  ways  (Belton,  1986:  10;  von  Winterfeldt  and  Edwards, 

1986:  275-276). 

Another  strength  of  VFT  is  the  ability  to  create  previously  unknown  new 
alternatives.  Keeney  observes  that  a  VFT  value  model  can  be  used  to  systematically 
probe  for  new  alternatives  that  may  be  better  than  those  that  are  first  identified  without 
systematic  analysis  (Keeney,  1994:  38-39).  The  value  model  eliminates  “anchoring”  to 
possibly  narrowly  defined  alternatives. 

Models  that  relate  projects  to  the  overall  economic  objectives  of  an  organization 
can  also  be  used  for  R&D  analysis.  Examples  are  traditional  cost  benefit  analysis,  rate  of 
return  analysis  and  risk  assessment.  These  models  are  extensions  of  standard  capital 
decision  making  economic  models  (Bretschneider,  1993:  129). 

Examples  of  Government  Multidimensional  Scoring  Studies 

An  example  of  a  major  AHP  study  is  “Implementing  an  Investment  Strategy  for 
CVX”  (Christian  and  Hacker,  1998).  This  recent  U.S.  Navy  analysis  is  being  used  in 
planning  a  new  aircraft  carrier  to  replace  the  current  Nimitz  class  ships.  Since  no 
research  and  development  for  an  aircraft  carrier  has  been  done  in  over  30  years,  the  Navy 
performed  the  study  to  determine  what  technologies  they  should  invest  in  to  build  into  the 
new  carrier  (Christian  and  Hacker,  1998:  1).  The  study  used  linear  programming 
techniques  to  optimize  the  benefits  of  its  programs  while  meeting  cost  constraints. 

Examples  of  major  VFT  studies  in  the  U.S.  Air  Force  are  the  Spacecast  2020 
study  (Burk  and  Parnell,  1997:  60-73)  and  the  Air  Force  2025  study  (Parnell  et.  al.,  1998: 
1336-1350).  Spacecast  2020  concentrated  on  space  technologies  needs  for  2020.  Air 
Force  2025  broadened  the  perspective  and  included  both  Air  and  Space  (Aerospace). 
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While  both  studies  are  landmarks  in  Air  Force  Strategic  planning,  they  do  not  fully 
address  how  to  develop  the  technologies  needed  to  make  future  weapon  systems  a  reality. 
The  studies  do  provide  starting  points  to  understand  values  for  technology  and  help  to 
identify  some  possible  measures. 

Golabi,  Kirkwood,  and  Sicherman  performed  a  study  for  the  Department  of 
Energy.  They  used  a  value  model  to  identify  the  technical  worth  of  R&D  solar  energy 
projects  to  score  in  response  to  a  Request  for  Proposal  (RFP).  The  value  model  measured 
the  technical  worth  of  the  proposed  projects.  A  mixed  integer  program  was  used  to 
identify  the  technical  worth  of  a  portfolio  of  programs.  Some  theoretical  work  was 
completed  allowing  the  value  of  individual  programs  to  be  added  together  to  find  the 
overall  value  of  a  portfolio.  The  constraints  for  the  math  program  considered  the 
“geographic  diversity  of  awards,  selection  of  redundant  or  complementary  efforts, 
funding  of  small  or  minority  business  proposals,  and  selection  of  proposals  including  cost 
sharing  provision”  (Golabi,  Kirkwood,  and  Sicherman,  1981:  176). 

A  very  recent  study  completed  by  Parnell  et.  al.,  conducted  an  analysis  for  the 
Operational  Support  Office  (OSO)  of  the  National  Reconnaissance  Office  (NRO)  using 
value  focused  thinking  to  identify  high  value  projects  that  support  strategic  objectives  and 
“select  the  best  portfolio  of  products  and  services  within  resources  and  programmatic 
constraints”  (Parnell  et.  ah,  1998:  1).  Their  process  was  to  develop  a  value  model  by 
interviewing  decision-makers  and  examining  relevant  documents.  They  used  the  value 
model  to  score  alternatives  and  then  used  a  mixed  integer  linear  program  to  optimize  the 
results. 
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Value  Focused  Thinking 

Fundamental  Objectives  Hierarchy  or  Value  Hierarchy  Theory 

According  to  Keeney,  "Values  are  what  we  care  about”  (Keeney,  1992:  3).  They 
are  the  reason  anyone  cares  about  a  decision.  Without  values,  anything  that  occurs  is 
acceptable  since  there  is  no  reason  to  care  about  the  consequences  of  a  decision’s 
outcome.  Alternatives  are  possible  actions,  which  will  bring  about  consequences. 

Values  measure  how  desirable  or  undesirable  an  alternative  is,  based  on  the  consequences 
an  alternative  will  bring  about.  "Value-focused  thinking  essentially  consists  of  two 
activities:  first  deciding  what  you  want  and  then  figuring  out  how  to  get  it"  (Keeney, 

1992:  3-4). 

Keeney  states  value  focused  thinking  is  not  always  used  by  decision-makers 
because  there  is  no  formal  process  to  use  the  method.  That  is,  "the  concepts  and 
procedures  have  not  been  developed  or  integrated  into  an  explicit  approach"  (Keeney, 
1992:  29).  While  the  decision-maker  does  use  values  to  guide  his  decisions,  they  are  not 
always  explicitly  expressed. 

The  decision  frame  for  a  decision  is  defined  by  its  decision  context  and  its 
fundamental  objectives.  The  decision  context  defines  the  alternatives  to  consider  for  a 
particular  decision  situation.  "The  fundamental  objectives  make  explicit  the  values  that 
one  cares  about  in  the  context  of  a  decision  and  define  the  class  of  consequences  of 
concern"  (Keeney,  1992:  30). 

Values  of  decision-makers  are  turned  into  objectives.  The  objectives  state  all  that 
is  important  in  a  decision  and  are  the  only  reasons  to  be  interested  in  a  decision.  The 
objectives  provide  guidance  for  action  and  are  the  basis  of  all  decision  analysis  modeling. 
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"An  objective  is  a  statement  of  something  that  one  desires  to  achieve.  It  is  characterized 
by  three  features:  a  decision  context,  an  object,  and  a  direction  of  preference"  (Keeney, 
1992:  33-34). 

Keeney  identifies  how  value  hierarchies  are  constructed.  Fundamental  objectives 
or  values  can  be  built  into  hierarchies.  Adjacent  objectives  or  objectives  at  the  same  level 
in  the  hierarchy  are  distinct  and  are  called  a  tier.  Lower  tier  objectives  are  part  of  the 
higher-tier  objective.  Higher-tier  objectives  are  defined  by  the  two  or  more  lower  tier 
objectives.  Adjacent  objectives  must  be  mutually  exclusive  and  collectively  exhaustive 
for  the  higher  tier  objective  (Keeney,  1992:  78).  The  highest  tier  of  the  hierarchy  is  the 
overall  fundamental  objective,  which  “characterizes  the  reason  for  interest  in  the  decision 
situation  and  defines  the  breadth  of  concern”  (Keeney,  1992:  77). 

Value  Hierarchy  Development  Methods 

There  are  several  different  accepted  ways  to  develop  value  hierarchies.  Kirkwood 
identifies  a  top  down  or  a  bottom  up  approach  as  possibilities  (Kirkwood,  1997:  19-23). 
The  top  down  approach  is  used  when  alternatives  are  not  well  specified.  The  values  are 
built  starting  at  the  highest  tier  and  then  broken  into  lower  and  lower  tiers.  Typically, 
information  for  this  method  comes  from  mission,  vision  and  strategic  documents.  The 
process  is  called  the  “Gold  Standard”  when  used  to  develop  a  value  hierarchy  (Parnell  et. 
al.,  1998:  1338).  If  access  to  decision-makers  in  charge  of  the  organization  is  possible, 
then  results  from  the  “Gold  Standard”  can  be  combined  with  the  decision-maker’s  inputs 
identified  through  interviews. 

In  the  bottom  up  approach  discussed  by  Kirkwood,  alternatives  are  known  and 
can  be  examined  to  determine  how  they  differ  from  each  other.  Grouping  the  differences 
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in  alternatives  together  into  higher  and  higher  tiers  forms  values  (Kirkwood,  1997:  19- 
23).  Another  approach  similar  to  Kirkwood’s  is  to  determine  what  tasks  the  organization 
perform  with  a  group  of  people  and  name  the  tasks  using  verbs.  The  words  can  be 
grouped  together  in  affinity  diagrams  and  then  structured  into  different  value  hierarchy 
tiers.  This  approach  is  called  the  “Silver  Standard”  (Parnell  et.  al.,  1998:  1340). 

In  all  approaches,  values  are  placed  in  the  hierarchy  only  if  using  the  value  to 
score  an  alternative  would  change  its  ranking.  Value  hierarchies  are  used  as  a  guide  for 
information  collection,  to  help  identify  alternatives,  to  facilitate  communications,  and  to 
evaluate  alternatives  (Kirkwood,  1997:  19-23). 

Measurement  Theory 

An  evaluation  measure,  sometimes  called  an  attribute,  measures  the  degree  to 
which  an  objective  or  a  value  has  been  obtained  (Keeney,  1992:  100;  Kirkwood,  1997: 
12).  While  a  fundamental  objective  measures  the  qualitative  aspects  of  a  decision, 
evaluation  measures  provide  quantitative  measures  for  the  decision.  A  value  hierarchy 
combined  with  evaluation  measures  is  a  value  model.  Generally,  a  value  model  uses 
many  quantities  to  measure  the  overall  fundamental  objective  in  the  hierarchy.  The 
evaluation  measure  converts  a  quantity  from  its  units  to  a  common  set  of  units  allowing 
the  many  attributes  to  be  combined  into  a  single  measure  of  merit  or  benefit.  Careful 
development  of  an  attribute  is  required  if  it  is  to  provide  correct  insight  to  the  decision¬ 
maker. 

Keeney  identifies  a  conceptual  part  and  a  measurement  part  called  a  scale  in 
measures.  Desirable  properties  for  evaluation  measures  are  it  should  be  measurable, 
understandable  and  operational  (Keeney,  1992:  112,  116). 
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Keeney  observes  measurable  attributes  define  objectives  in  more  detail  than  the 
objective  by  itself.  The  attribute  embodies  appropriate  implicit  value  judgements  and 
avoids  inappropriate  judgements.  It  measures  what  the  decision-maker  is  interested  in 
and  does  not  measure  other  items.  Ambiguity  can  occur  if  the  levels  of  the  attribute  are 
not  well  defined  (Keeney,  1992:  1 13).  The  clairvoyance  test  should  be  applied  to 
determine  if  the  measure  is  ambiguous  or  not.  The  test  is  “if  a  clairvoyant  were  available 
who  could  foresee  the  future  with  no  uncertainty,  would  this  clairvoyant  be  able  to 
unambiguously  assign  a  score  to  the  outcome  from  each  alternative  in  a  decision 
problem"  (Kirkwood,  1997:  28)? 

An  understandable  evaluation  measure  is  implied  by  the  requirements  for  a 
measurable  evaluation  measure.  If  one  person  assigns  a  level  to  an  attribute,  another 
should  be  able  to  interpret  the  level  with  no  loss  of  information  (Keeney,  1992:  1 16).  If 
the  evaluation  measure  is  ambiguous,  it  will  not  be  understandable. 

Keeney  defines  a  measure  as  operational  if  it  describes  possible  consequences 
with  respect  to  the  associated  objective  and  it  provides  "a  sound  basis  for  value 
judgements  about  the  desirability  of  the  various  degrees  to  which  the  objective  might  be 
achieved."  Besides  defining  a  measure,  other  information  that  should  be  considered  is 
where  it  is  measured,  how  often  it  is  measured,  or  how  multiple  measures  aggregate  over 
time  or  space.  It  is  preferred  that  a  measure  be  conditionally  independent  of  all  of  the 
other  measures  in  a  hierarchy  (Keeney,  1992:  1 14-1 15). 

Kirkwood  identifies  four  different  types  of  scales  for  measuring  values  in  value 
hierarchies.  A  scale  can  be  either  natural  or  constructed  and  can  be  either  a  direct  scale 
or  a  proxy  measure.  Natural  scales  are  those  in  general  use  that  have  a  common 
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interpretation.  Constructed  scales  are  developed  specially  for  a  particular  measure. 

Direct  scales  specifically  measure  the  quantity  being  examined  while  proxy  scales  are 
correlated  with  the  quantity  being  measure,  but  actually  measure  something  different  then 
the  value  being  quantified  (Kirkwood,  1997:  24). 

Keeney  identified  some  of  the  advantages  and  disadvantages  with  constructed 
attributes  or  evaluation  measures.  Constructed  attributes  exactly  measure  what  the 
fundamental  objective  is  meant  to  address.  They  completely  describe  the  objective.  It  is 
simpler  to  separate  consequences  from  value  judgements.  Problems  with  constructed 
measures  include  being  understandable  and  operational.  The  consequences  of 
constructed  attributes  are  more  difficult  to  communicate  since  they  do  not  have  a 
common  interpretation  (Keeney,  1992:  118-119). 

Keeney  also  provides  an  excellent  discussion  on  proxy  measures.  Proxy  measures 
reduce  the  number  of  attributes  needed  for  a  decision  and  simplify  descriptions  of  the 
consequences.  The  effort  to  gather  factual  information  is  reduced,  but  the  effort  to 
specify  the  value  model  increases.  Issues  of  fact  are  combined  with  issues  of  value, 
which  results  in  asking  technical  experts  to  make  value  judgements  and  results  in  less 
insight  (Keeney,  1992:  119-121). 

Measurement  Scales  and  Single  Dimension  Value  Functions 

Luce,  Bush  and  Galanter  identify  several  scale  classifications  (Luce,  Bush  and 
Galanter,  1963:  8-13).  In  order  of  decreasing  information,  scales  can  be  absolute,  ratio, 
interval,  ordinal,  and  nominal  scales.  The  difference  scale,  is  a  special  case  of  the 
interval  scale.  Additionally  each  scale  has  an  “allowable”  transformation  that  does  not 
change  the  scale  classification. 
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Scale  Type 

Definition 

Allowable 

Transformation 

Example 

Absolute 

Units  and  Origin  Fixed 

Multiplication  by 
Identity 

Counting 

Ratio 

Units  Arbitrary,  Origin  Fixed 

Multiplication  by 

Scalar 

Mass  (grams, 
pounds,  etc.) 

Interval 

Units  and  Origin  Arbitrary 

Positive  Linear 
(a*x  +  b) 

Fahrenheit  or 
Celsius 

Ordinal 

Only  order  matters 

Monotonic  increasing 
or  decreasing  functions 

Moh’s 

hardness  scale 
for  minerals 

Nominal 

Arbitrary 

Any  one  to  one 

National  Draft 
numbers  or 
football  player 
numbers. 

Table  2:  Summary  of  Scale  types  (Luce,  Bush  and  Galanter,  1963:  8-13) 


Luce,  Bush  and  Galanter  state  that  some  scales  are  defined  based  on  empirical 
operations.  Defining  scales  this  way  “permits  the  subject  (observer  or  experimenter)  to 
compare  intervals  and  to  indicate  in  some  way  whether  or  not  they  are  equal”  (Luce, 
Bush,  and  Galanter,  1963:  15).  By  defining  the  scale  in  terms  of  a  linear  transformation, 
an  interval  scale  is  created. 

Roberts  provides  examples  of  meaningful  and  meaningless  statements  due  to 
scale  type.  He  determines  it  is  meaningful  to  say  f(a)=2f(b),  where  the  function  f 
transforms  the  quantities  a  and  b,  when  the  scale  of  f  is  a  ratio  scale.  Similarly,  it  is  also 
meaningful  to  say  f(a)/f(b)  =  X,  where  X  is  a  constant.  These  statements  are  meaningless, 
however,  if  the  scale  of  f  is  an  interval  scale.  Another  example  of  a  meaningful  statement 
is  to  state  f(a)  -  f(b)>f(c)  -  f(d),  where  the  function  f  transforms  the  quantities  a,  b,  c  and 
d,  when  the  scale  of  f  is  an  interval  scale  (Roberts,  1979:  71-74). 

In  decisions  under  certainty,  three  types  of  functions  are  often  used  to  define  an 
attribute.  These  are  discrete  functions  (Clemen,  1996:  BO),  exponential  functions 
(Kirkwood,  1997:  62-64;  Keeney,  1992:  141-146)  and  piecewise  linear  functions 
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(Kirkwood,  1997:  64-68).  If  proper  value  increment  questions  are  asked,  these 
evaluation  measures  will  transform  a  raw  measurement  to  an  interval  scale  (von 
Winterfeldt  and  Edwards,  1986:  209-210). 

The  single  dimension  value  function  converts  a  quantity  being  measured  into 
value.  Value  is  typically  measured  between  0  and  1  (Kirkwood,  1997:  61),  but  can  be 
measured  from  0  to  10  and  0  to  100.  Which  scale  is  used  for  analysis  has  no  effect  on  the 
analysis  results  as  long  as  one  scale  is  used  for  all  measures  in  the  value  model  and  if  an 
additive  value  function  (discussed  later)  is  used. 

Finding  Alternatives  and  Attribute  Scoring 

Air  Vehicles  has  already  developed  alternatives,  which  can  be  scored  using  the 
VA  value  model  (UAV,  Sustainment  and  Trans-Atmospheric  &  Space  Integrating 
Concept  Roadmaps,  1999).  However,  with  a  value  model,  it  may  be  possible  to  improve 
the  existing  alternatives  or  identify  new  alternatives.  Keeney  notes,  the  fundamental 
objectives  hierarchy  or  value  hierarchy  lists  all  that  is  important  to  a  decision  context. 
Since  this  is  true,  it  is  possible  to  use  the  value  hierarchy  to  identify  better  alternatives. 
This  can  be  accomplished  by  improving  existing  alternatives  to  score  higher  using  the 
model  or  by  identifying  new  alternatives  which  score  better  (Keeney,  1992:  201-202). 

Kirkwood  identifies  possible  methods  of  improving  or  finding  new  alternatives. 
He  suggests  considering  each  evaluation  measure  one  at  a  time  and  identifying  ways  to 
improve  the  alternative  in  that  particular  area.  It  may  be  that  the  alternative  is  not 
attractive  by  improving  it  in  a  single  area,  but  the  exercise  can  suggest  other  attractive 
alternatives.  Another  approach  is  to  improve  several  evaluation  measures  under  a  single 
value  for  an  alternative  to  “maximize”  a  particular  value.  Finally,  it  may  be  possible  to 
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develop  alternatives  that  directly  balance  all  of  the  evaluation  measures  (Kirkwood,  1997: 
49-50). 

Scoring  alternatives  is  straight  forward,  but  can  be  time  consuming.  Technical 
experts  consider  each  alternative  one  at  a  time  for  each  attribute.  In  the  case  of  multiple 
technical  experts,  a  consensus  should  be  reached  about  how  the  alternative  scores  in  the 
area  under  consideration. 

The  only  scores  allowed  are  defined  by  the  evaluation  measures.  Each  evaluation 
measure  converts  the  alternative  score  to  value.  The  value  scores  for  each  attribute  are 
then  combined  using  value  model  weights  and  the  overall  value  model  function  resulting 
in  a  single  measure  of  merit  for  the  overall  fundamental  objective  being  considered  for 
the  decision.  Kirkwood  provides  an  exponential  function  and  a  piecewise  linear  function 
written  in  Microsoft  Excel  Visual  Basic  for  converting  raw  scores  to  value  (Kirkwood, 
1997:  78-81).  He  also  provides  detailed  implementation  methodology  for  evaluating 
value  models  in  Excel  (Kirkwood,  1997:  75-81). 

Assessing  Weights 

The  method  of  swing  weights  is  commonly  used  to  assess  the  weights  for  the 
values  in  a  hierarchy  although  other  methods  such  as  pricing  out  and  lottery  weights  are 
possible  (Clemen  1996:  546-552).  The  swing  weight  method  is  a  thought  experiment 
where  “the  decision  maker  compares  individual  attributes  directly  by  imagining 
(typically)  hypothetical  outcomes”  (Clemen  1996:  547). 

Kirkwood  discusses  the  method  in  depth  and  this  information  on  weights  is  based 
on  that  discussion  (Kirkwood,  1997:  68-70).  Each  tier  of  values  or  measures  in  the 
hierarchy  is  considered  individually.  It  is  important  to  consider  the  ranges  for  measures 
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since  changes  in  the  ranges  of  the  measure  can  change  the  rankings  of  the  alternatives 
(Kirkwood,  1997:  58-59;  von  Winterfeldt  and  Edwards,  1986:  285,  368).  The  decision¬ 
maker  or  the  decision-maker’s  experts  are  asked  to  rank  order  the  values  or  measures 
from  least  important  to  most  important.  The  ranges  for  each  value  or  measure  are  then 
examined  one  at  a  time.  The  decision-maker  is  asked  to  imagine  the  value  or  measure 
changing  from  its  lowest  to  its  highest  value  for  each  of  the  values  or  measures. 

The  decision-maker  or  the  decision-maker’s  experts  are  then  asked,  “How  much 
better  is  an  increase  (decrease)  in  value  of  value  A  to  value  B  considering  the  ranges  A 
and  B  can  cover?”  The  process  continues  until  there  is  one  less  equation  then  the  total 
number  of  categories.  The  last  equation  needed  is  to  sum  the  values  for  all  categories  to 
one.  The  result  is  a  system  of  equations  with  the  same  number  of  equations  and 
unknowns,  which  is  easily  solved. 

For  the  highest  tier  values,  this  process  can  be  complex  since  understanding  the 
highest  and  lowest  levels  a  value  can  take  on  is  dependent  on  all  of  the  values  and 
measures  below  it.  This  can  result  in  a  large  number  of  variables  that  must  be  considered 
while  determining  the  weights  (Keeney  and  Raifa,  1976:  125). 

Multiobjective  Additive  Value  Functions 

The  next  step,  once  scoring  of  alternatives  and  assessing  weights  is  completed,  is 
to  combine  all  of  the  scores  into  a  single  measure  of  merit.  The  most  straightforward 
way  of  accomplishing  this  is  to  use  an  additive  value  function  although  others  such  as 
additive  linear,  multiplicative  and  multilinear  are  possible  (von  Winterfeldt  and  Edwards, 
1986:  275-276).  The  additive  multiobjective  value  function  is  defined  as: 

n 

v(x)  =  X  wivi  )  Equation  2 

;=i 
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where  x  is  the  overall  evaluation  objective,  Xj  is  the  raw  score  of  attribute  i,  v,  is  the 
single  dimension  value  function,  and  n  is  the  total  number  of  evaluation  measures. 

Keeney,  Kirkwood  and  Clemen  all  discuss  necessary  and  sufficient  conditions  to 
use  an  additive  value  function  with  value  hierarchies  (Keeney,  1992:  132-138;  Kirkwood, 
1997:  238-239;  Clemen,  1996:  579-580).  Mutual  preferential  independence  of  attributes 
or  evaluation  measures  is  required  to  use  an  additive  value  function  under  conditions  of 
certainty.  According  to  Clemen,  “An  attribute  Y  is  said  to  [be]  preferentially 
independent  of  X  if  preferences  for  specific  outcomes  of  Y  do  not  depend  on  the  level  of 
attribute  X”  (Clemen,  1996:  579).  Mutual  preferentially  independent  means  Y  is 
preferentially  independent  of  X  and  X  is  preferentially  independent  of  Y. 

Clemen  maintains  mutual  preferential  independence  holds  “for  many  people  and 
many  situations,  or  that  at  least  it  is  a  reasonable  approximation”  (Clemen,  1996:  579). 

He  relates  this  to  the  concept  of  decomposability,  which  is  the  ability  to  break  down 
higher  tier  values  into  a  series  of  lower  tier  values.  Clemen  states,  “If  a  decision  maker 
has  done  a  good  job  of  building  a  decomposable  hierarchy,  mutual  preferential 
independence  probably  is  a  reasonable  assumption.  But  it  should  never  be  taken  for 
granted”  (Clemen,  1996:  579).  Both  Clemen  and  Kirkwood  provide  formal  testing 
procedures  (Clemen,  1996:  580-582;  Kirkwood,  1997:  238-239). 

Fischer  studied  the  suitability  of  using  additive  and  multiplicative  value  and 
utility  functions  in  multi-attribute  utility  assessments.  He  identifies  risky  and  riskless 
functions  where  a  decision  is  riskless  “if  the  decision  maker  can  specify  with  certainty  the 
outcome  which  will  results  from  each  course  of  action  under  consideration”  (Fischer, 
1977:  297).  A  decision  is  risky  “if  the  [decision-maker]  is  uncertain  about  the  outcome 
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of  any  given  action,  but  is  able  to  express  this  uncertainty  in  the  form  of  a  subjective 
probability  distribution  over  the  set  of  all  possible  outcomes”  (Fischer,  1977:  299). 

Fischer  experimented  by  comparing  results  from  small  value  hierarchies  with 
holistic  (direct  assignment  of  overall-all  value  to  outcomes  by  the  decision-maker) 
results.  He  examined  risky  and  riskless  value  functions  and  he  examined  use  of  additive 
and  multiplicative  value  functions.  Fischer  found  high  correlations  between  all  of  the 
functions  in  his  tests  with  a  majority  of  the  ten  people  he  worked  with.  Fischer’s  data 
shows  the  additive  value  function  provides  a  good  model  of  decision-makers  preferences, 
but  there  are  some  rare  times  it  does  not  work.  He  does  not  recommend  ignoring 
differences  between  risky  and  riskless  decisions  and  additive  and  multiplicative  value 
functions  (Fischer,  1977:  313-314). 

Ranking  of  Results 

Kirkwood  provides  information  on  how  to  rank  results.  Once  all  of  the 
alternatives  are  scored  and  the  additive  value  function  is  implemented  to  identify  the 
overall  score  for  each  alternative,  it  is  a  simple  matter  to  rank  order  the  alternatives.  It  is 
also  possible  to  develop  graphs  that  document  the  contribution  made  to  each  value  in  the 
hierarchy.  The  graphs  provide  insight  on  why  a  particular  alternative  scored  the  way  it 
did  and  can  be  the  basis  for  improving  an  alternative.  Kirkwood  provides 
implementation  methods  to  determine  the  contribution  each  value  makes  to  the  scoring  of 
an  alternative  (Kirkwood,  1996:  76-81) 

Sensitivity  Analysis 

Sensitivity  analysis  on  the  weights  can  be  performed  to  determine  the  weighting 
levels  that  will  change  the  alternative  rankings  and  can  determine  what  the  changes  in 
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policy  would  be.  If  the  procedure  is  insensitive  to  meaningful  variations  in  the  weights, 

further  discussion  is  not  necessary.  If  particular  weightings  are  sensitive,  analysis  of  the 

weights  can  be  focused  on  the  specific  alternatives  affected  (Kirkwood,  1997:  82-85). 

Sensitivity  analysis  is  performed  by  changing  the  weight  of  a  single  evaluation 

measure  or  a  single  value  while  holding  the  other  weights  to  the  same  ratio  as  the  base 

case.  All  of  the  weights  for  an  evaluation  measure  tier  or  value  tier  must  add  up  to  1. 

Hence,  a  particular  weight  is  varied  from  0  to  1.  Kirkwood  identifies  the  method  of 

varying  all  other  weights  in  a  particular  tier  as  follows: 

For  each  weight  except  the  one  that  you  are  varying,  the  equation  is  given 
by  the  following:  Multiply  1  minus  the  weight  that  is  being  varied  by  the 
ratio  of  the  base  case  value  for  the  weight  being  considered  to  the  sum  of 
the  base  case  weights  for  all  the  weights  except  the  one  that  is  being 
manually  varied  (Kirkwood,  1997:  84). 

He  also  provides  simple  examples  for  implementing  sensitivity  analysis  in  a  Microsoft 
Excel  spreadsheet  (Kirkwood,  1997:  82-85). 

Resource  Allocation 

A  problem  faced  by  R&D  organizations  is  determining  how  to  allocate  scarce 

funds  to  a  large  number  of  competing  projects.  Kirkwood  observes: 

. . .  when  there  is  a  specified  budget  to  be  allocated  among  competing 
activities,  then  the  question  of  how  the  resources  not  allocated  to  a 
particular  activity  are  used  becomes  more  important  in  evaluating 
alternatives  since  these  resources  are  available  to  fund  other  activities 
(Kirkwood,  1997:199). 

The  problem  is  a  question  of  identifying  a  group  of  projects  or  programs,  which  provide 
maximum  “value”  for  the  time,  resources  or  dollars  invested.  Kirkwood  identifies 
benefit/cost  analysis  and  optimization  methods  for  solving  the  resource  allocation 
problem  (Kirkwood,  1996:  199). 


33 


The  process  is  to  calculate  the  ratio  of  the  benefit  of  project  (determined  from  a 
value  model)  with  the  cost  of  funding  the  project.  The  ratio  provides  insight  into  the 
projects  that  provide  the  most  benefit  per  dollar  invested.  By  sorting  the  projects  in 
ascending  order  based  on  the  ratio,  a  funding  order  can  be  found.  The  cumulative  benefit 
is  then  graphed  against  the  cumulative  cost.  The  method  assumes  the  value  and  cost  of  a 
project  does  not  change  based  on  which  project  is  being  funded.  Additionally,  the 
number  of  constraints  allowed  by  the  analysis  is  limited  to  the  single  cost  constraint. 
Another  concern  with  the  method  is  it  may  be  possible  to  choose  a  different  combination 
of  projects,  which  result  in  an  equal  or  higher  value  and  more  fully  use  the  budget.  In 
other  words,  an  optimal  portfolio  is  not  guaranteed  (Kirkwood,  1996:  200-205;  Bell  and 
Read,  1974:  36). 

Bretschneider  notes  constrained  optimization  models  have  a  set  of 
equations  containing  decision  variables  called  constraints.  The  model  contains  the 
objective  function,  constraints  and  decision  variables.  Linear  Programming,  integer 
programming  and  nonlinear  programming  are  typical  optimization  models  used  in  R&D 
decision  analysis  (Bretschneider,  1993:  130).  Assumptions  for  linear  programming 
techniques  can  be  found  in  any  linear  programming  text  and  are  summarized  in  Table  3 
(Winston,  1994:  53-54). 

It  is  possible  to  use  a  linear  model  where  the  decision  variables  are  the  extent  of 
resource  commitment  to  a  project  (Bretschneider,  1993:  130)  or  to  use  integer 
programming  where  the  decision  is  whether  or  not  to  include  a  particular  program. 
Typical  constraint  types  are  budget  constraints,  skill  availability  constraints,  facility 
availability  constraints,  raw  material  availability  constraints,  and  program  balance  and/or 


34 


risk  balance  constraints  (Bretschneider,  1993:  130;  Cetron,  Martins,  and  Roepoke,  1967, 
5). 


Proportionality 

Assumption 

The  contribution  of  the  objective  function  from  each  decision  variable 
is  proportional  to  the  value  of  the  decision  variable.  The  contribution 
of  each  variable  to  the  left-hand  side  of  each  constraint  is  proportional 
to  the  value  of  the  variable. 

Additivity 

Assumption 

The  contribution  of  the  objective  function  for  any  variable  is 
independent  of  the  values  of  the  other  decision  variables.  The 
contribution  of  a  variable  to  the  left-hand  side  of  each  constraint  is 
independent  of  the  values  of  the  variable. 

Divisibility 

Assumption 

Requires  each  decision  variable  be  allowed  to  assume  fractional 
values.  If  this  does  not  hold,  than  integer  programming  should  be 
used. 

n 

Each  parameter  (objective  function  coefficient,  right  hand  side,  and 
technological  coefficient)  is  known  with  certainty. 

Table  3:  Linear  Programming  Assumptions  (Winston,  1994:  53-54) 


Many  types  of  optimization  R&D  portfolio  selection  models  have  been  suggested 
(Gear,  Lockett,  and  Pearson,  1971 :  66-76).  Of  the  wide  array  of  possible  models 
discussed,  the  linear  programming  approach  used  by  Bell  and  Read  handles  optimization 
of  scarce  resources  to  maximize  the  benefit  of  research  and  development  programs  (Bell 
and  Read,  1970:  35-42).  The  decision  variables  can  be  restricted  to  integer  solutions  only. 
In  his  1974  article,  Gear  notes  Bell’s  model  has  been  tested,  allows  multiple  resource 
constraints  and  allows  multiple  planning  periods.  It  assumes  1)  a  selected  project  is 
continued  until  completion  and  2)  perfect  knowledge  of  required  resources.  Additionally, 
it  assumes  one  objective  involved  in  the  resource  allocation  problem  (Gear,  1974:  199- 
120). 


Mixed  integer  programs  methods  exist  for  expressing  interactions  between 
projects  (Aaker  and  Tyebjee,  1978;  Schmidt,  1993).  Aaker  and  Tyebjee  develop  a  model 
able  to  handle  overlap  in  project  resource  utilization,  technical  project  interdependence, 
and  project  interaction  with  respect  to  value  contribution.  Schmidt  developed  a  nonlinear 
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integer  program  which  can  handle  benefit  interactions,  resource  interactions,  and 
outcome  interactions. 

Besides  restricting  the  decision  variables  to  integer  solutions  only,  constraints  will 
be  required  when  selection  of  one  program,  (say  program  A),  requires  several  other 
programs,  (say  programs  B,  C,  and  D)  to  be  selected.  At  the  same  time,  program  B 
required  to  support  program  A  might  be  needed  to  support  another  program  (say  program 
E).  Constraints  are  required  to  allow  program  B  to  be  selected  if  either  programs  A,  E  or 
both  A  and  E  are  selected.  If  neither  program  A  or  E  is  selected,  than  program  B  should 
not  be  selected.  Williams  provides  a  good  discussion  of  integer  programming  and  logical 
constraints  to  help  formulate  this  and  other  types  of  restrictions  (Williams,  1985:  148- 
225). 

R&D  Model  Characteristics  and  Known  Limitations 

Cetron,  Martino  and  Roepcke  surveyed  R&D  selection  models  and  identified  a 
number  of  features  and  ease  of  use  considerations,  which  are  shown  in  Table  4. 
Bretschneider  and  Sounder  have  developed  criteria  for  determining  the  aptness  of  an 
R&D  model.  Table  5  shows  the  criteria  originally  developed  by  Sounder  and  updated  by 
Bretschneider. 
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Criteria 

Concept 

Utility  Measure 

Does  the  method  use  a  method  to  measure  the  utility  or  value  of 
success  of  an  R&D  project  (i.e.  market  share,  profitability,  military 
worth) 

Probability  of 
Success 

Does  the  model  explicitly  measure  the  probability  of  success  of  each 
R&D  project? 

Orthogonality 
of  Criteria 

Are  the  criteria  used  for  evaluation  mutually  exclusive  or  orthogonal? 

Sensitivity 

Has  sensitivity  of  output  to  small  changes  in  input  been  checked?  A 
large  change  in  output  to  small  changes  in  input  is  not  desirable. 

Rejected 

Alternatives 

Retention 

If  a  project  is  rejected  for  funding,  can  it  be  included  later  if  funding 
changes? 

Classification 

Structure 

Does  the  method  provide  a  structural  relationship  between  the  project 
and  higher  level  organizational  goals? 

Time 

Does  the  method  include  schedule  requirements  or  provide  scheduling 
outputs? 

Strategies 

Does  the  method  provide  a  method  to  take  into  account  several 
possible  scenarios,  world  environments,  market  situations,  etc.? 

System  Cross 
Support 

Does  the  method  give  system  development  credit  for  support  which  it 
provides  to  another  system  development? 

Technology 

Cross  Support 

Does  the  method  give  a  project  for  advancement  of  technology  credit 
for  support  which  it  provides  to  the  advancement  of  other 
technologies? 

Graphical 

Display 

Can  the  results  be  displayed  graphically  allowing  fast  evaluation  of 
various  projects? 

Flagging 

Does  the  method  identify  problem  areas  allowing  user  intervention? 

Optimization 

Criteria 

What  is  the  optimization  method  and  what  constraints  are  considered? 

Constraints 

In  the  review  the  following  constraints  were  considered:  budget,  skills 
available,  facilities  available,  competitor  efforts,  and  raw  materials 
available 

Computerized 

Is  the  method  computerized? 

Data 

Requirements 

More  data  provides  better  information  on  the  results,  but  is  more 
difficult  to  collect.  Considerations  are  the  level  of  organization  the 
data  is  collected  at  and  the  amount  of  information  needed. 

Manual 

Can  the  method  be  applied  manually  or  is  a  computer  required? 

Computer 

Program 

Has  the  method  been  automated  on  a  computer? 

How  long  does  it  take  to  run  the  model  and  get  results? 

How  easy  is  it  to  update  the  model  with  new  information? 

What  skill  level  is  needed  by  the  operator  to  run  the  model? 

Does  the  R&D  organization  require  outside  help  to  use  the  method? 

Table  4:  Model  Features  and  Ease  of  Use  Considerations  (Cetron,  Martino 

and  Ropcke,  1 967:  4-5) 
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Realism  Criterion 

Capability  Criterion 

Model  Includes: 

Multiple  objective 

Multiple  constraints 

Market  risk 

Technical  risk 

Manpower  limits 

Facility  limits 

Budget  limits 

Premises  uncertainty 
Multiple  time  periods 
Hierarchical  structure 
Competitor  effects 

Model  Performs: 

Multiple  time  period  analysis 
Optimization  analysis 
Simulation  analysis 

Scheduling  analysis 

Flexibility  Criterion 

Use  Criterion 

Model  applicable  to: 
Applied  projects 

Basic  projects 

Priority  decisions 
Termination  decisions 
Budget  allocation 

Project  funding 

Model  characterized  by: 
Familiar  variables 

Discrete  variables 

Computer  not  needed 

Special  person  not  needed 
Special  interpretations  not 
needed 

Low  amount  of  data  needed 
Easy  to  obtain 

Cost  Criterion 

Additional  Criterion 

Model  has: 

Low  set  up  costs 

Low  personnel  costs 

Low  computer  costs 

Low  data  collection  costs 

Model  considers: 

Predictive  accuracy 

Strategy  needs 

Project  interdependencies 

Table  5:  Model  Evaluation  Criteria  (Bretschneider,  1993: 135;  Sounder, 

1972:  B-528) 

Baker  and  Freeland  note  seven  limitations  of  benefit  measurement  models  and  resource 
allocation  models: 

1)  Inadequate  treatment  of  risk  and  uncertainty. 

2)  Inadequate  treatment  of  multiple,  often  interrelated,  criteria. 

3)  Inadequate  treatment  of  project  interrelationships  with  respect  both  to  value 
contribution  and  to  resource  utilization. 

4)  No  explicit  recognition  and  incorporation  of  the  experience  and  knowledge  of 
the  R  and  D  manager, 

5)  The  inability  to  recognize  and  treat  nonmonetary  aspects  such  as  establishing 
and  maintaining  balance  of  R  and  D  program;  (e.g.,  balance  between  basic 
and  applied  work,  between  offensive  and  defensive  activity,  between  product 
and  process  effort,  between  in-house  and  contracted  projects,  between 
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improvement  and  break-through  orientation,  and  between  high  risk-high 
payoff  and  moderate  or  low  risk-moderate  payoff  opportunities). 

6)  Perceptions  held  by  the  R&D  managers  that  the  models  are  unnecessarily 
difficult  to  understand  and  use. 

7)  Inadequate  treatment  of  the  time  variant  property  of  data  and  criteria  and  the 
associated  problem  of  consistency  in  the  research  program  and  the  research 
staff  (Baker  and  Freeland,  1975:  1 165). 

Baker  and  Freeland’s  criteria  identify  properties  R&D  models  should  have  or  they  will 
not  completely  cover  what  is  important. 

Literature  Review  Summary 

This  chapter  has  identified  the  dimensions  of  aerospace  R&D  and  those  specific 
to  Air  Vehicles’  needs.  It  has  reviewed  technology  selection  models  and  identified  some 
previous  studies  that  use  VFT.  It  has  provided  an  in-depth  review  of  value  focused 
thinking  and  resource  allocation  models.  Finally,  the  R&D  modeling  criteria  and 
limitations  of  previous  R&D  selection  models  were  examined. 
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Chapter  3:  Methodology  and  Modeling 

Introduction 

The  overall  goal  for  this  project  is  to  identify  the  technological  direction  Air 
Vehicles  should  pursue  to  be  most  supportive  of  USAF  values.  Air  Vehicles  sets  its 
direction  by  selecting,  maintaining,  and  funding  new  technology  programs.  These 
programs  fall  under  the  categories  of  basic  research,  applied  research,  or  development 
research.  Working  from  the  foundation  laid  in  the  literature,  a  methodology  to  evaluate 
Air  Vehicles’  R&D  technology  selection  was  developed. 

The  first  topic  covered  in  this  chapter  is  an  overview  of  developing  the  Air 
Vehicles  model.  Reasons  for  selecting  value  focused  thinking  with  optimization  to 
answer  Air  Vehicles’  challenge  and  an  overview  of  the  analysis  process  are  provided. 

The  next  topic  covered  is  the  development  of  the  Air  Vehicles’  value  hierarchy.  The 
method  of  developing  the  value  hierarchy  is  discussed  followed  by  literature  search 
integration,  which  allowed  development  of  a  notional  hierarchy.  The  final  hierarchy 
created  by  Air  Vehicles’  experts  from  the  notional  model  is  also  presented. 

The  third  topic  covered  is  development  of  the  Air  Vehicles’  value  model.  This 
topic  covers  considerations  of  measuring  technological  change,  development  of 
evaluation  measures,  aggregation  of  individual  technology  measurements  using  the 
additive  value  function,  and  weighting  of  values  and  evaluation  measures.  The  last  topics 
covered  are  modeling  assumptions,  applying  the  Air  Vehicles’  value  model,  and  an 
exploration  of  how  suitable  the  Air  Vehicles’  complete  model  is  based  on  the  literature. 
The  latter  topic  checks  that  the  model  is  collectively  exhaustive,  compares  the  model  with 
criteria  from  previous  efforts,  and  searches  for  major  limitations. 
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Air  Vehicles’  Model  Development  Overview 

Model  Selection 

In  the  literature,  different  types  of  R&D  technology  selection  models  are 
discussed.  Essentially,  Air  Vehicles  wants  to  maximize  the  benefit  technology  provides 
to  the  warfighter  while  minimizing  the  investment  costs  required  to  achieve  that  benefit. 
As  reviewed  in  Chapter  2,  there  are  three  types  of  models  that  might  be  applicable  for 
identifying  benefit:  comparative  techniques,  multidimensional  scoring,  and  extensions  of 
economic  models  (Bretschneider,  1993:  124;  Baker  and  Freeland,  1975:  1164). 

The  comparative  models  are  often  inconsistent  and  nearly  impossible  to  aggregate 
results  (Bretschneider,  1993:  127-128).  Because  Air  Vehicles  has  over  100  programs, 
these  problems  are  magnified.  It  was  not  practical  to  get  every  program  manager 
together  in  a  group  to  perform  comparisons.  The  difficulties  in  aggregating  the  results 
could  prevent  comparisons  of  the  benefit  versus  the  cost  of  technologies.  The  extensions 
of  economic  models  were  also  dismissed,  since  R&D  requires  analysis  of  non-numeric 
concepts  such  as  revolutionary  versus  evolutionary  technology  and  programmatic  risk. 
Converting  these  concepts  to  monetary  value  adds  complexity  to  a  model  and  possibly 
adds  error  because  of  the  complexity. 

Since  R&D  is  multidimensional  (see  Table  1  in  Chapter  2  for  literature 
references),  multidimensional  models  were  deemed  the  most  appropriate  approach. 
Multidimensional  models  have  been  used  for  technology  selection  problems  in  the  past 
(Golabi,  Kirkwood,  and  Sicherman,  1981).  Of  the  two  types  of  multidimensional  models 
discussed  (value  focused  thinking  and  analytical  hierarchy  process),  value  focused 
thinking  is  the  best  fit.  As  will  be  shown  in  Chapter  4,  it  handles  most  of  Baker  and 
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Freeland’s  concerns  by  being  able  to  treat  risk,  multicriteria,  interrelated  criteria,  value 
contribution,  inputs  from  experienced  R&D  experts,  time  variant  data  and  nonmonetary 
aspects  of  R&D  (Baker  and  Freeland,  1975:  1 165).  Further,  value  focused  thinking, 
when  properly  applied,  does  not  have  the  theoretical  problems  and  limitations  known  to 
exist  in  the  analytical  hierarchy  process  (Belton,  1986:  10,18).  It  allows  creative 
development  of  new  alternatives  (Keeney,  1994:  38-39)  and  does  not  require  partial  re- 
evaluation  of  all  its  alternatives  when  a  new  alternative  is  added  as  comparative  methods 
do  (Bretschneider,  1993:  127-128). 

For  resource  allocation  or  identifying  the  best  benefit  at  the  lowest  cost, 
Bretschneider  identified  1)  constrained  optimization  models,  2)  emulations  of 
organizational  and  human  processes  (simulation),  or  3)  Ad-hoc  in  nature  models 
(Bretschneider,  1993:  124).  The  constrained  optimization  models  (Kirkwood,  1997:206- 
216;  Bell  and  Read,  1974;  Gear,  Lockett  and  Pearson,  1971;  Aaker  and  Tyebjee,  1978; 
Schmidt,  1993)  and  calculations  of  benefit  to  cost  ratio  are  the  most  appropriate 
(Kirkwood,  1997:  199-206;  Bell  and  Read,  1974:  36).  Simulation  is  not  practical  since  it 
is  impossible  to  meaningfully  model  all  of  the  outcomes  of  a  single  R&D  program,  much 
less  hundreds  of  programs.  This  condition  is  further  exacerbated  when  attempting  to 
simulate  revolutionary  programs,  which  have  limited  data  available  on  which  to  base  a 
model.  “Ad-hoc”  models  were  eliminated  from  consideration  since  these  models 
concentrate  on  using  regression  for  decision  making.  The  problem  with  these  type  of 
models  is  the  inability  to  make  new  decisions  not  made  in  the  past  (Bretschneider,  1993: 
131). 
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Analysis  Process 

The  framework  identifying  analysis  steps  needed  for  program  selection  in  Air 
Vehicles  is  shown  in  Figure  1.  The  first  step  in  the  study  was  to  identify  a  value 
hierarchy  for  VA.  The  R&D  literature,  summarized  in  Table  1  on  page  17,  is  the  starting 
point  to  determine  the  overall  fundamental  objective  for  the  decision  and  to  determine  the 
intermediate  objectives  supporting  the  overall  objective.  Since  the  model  must  represent 
VA’s  values,  VA  strategic  planning  experts  and  integrated  concept  leaders  were 
consulted  to  confirm,  modify,  and  develop  value  definitions.  The  value  hierarchy  was 
next  approved  by  the  Air  Vehicles  Directorate  senior  leaders. 
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came  to  general  agreement  on  the  shape  of  these  functions.  The  evaluation  measures 
combined  with  the  value  hierarchy  and  weights  define  the  value  model. 

After  the  Air  Vehicles  senior  management  approved  the  value  model,  alternatives 
or  VA  R&D  programs  were  examined.  The  starting  place  for  identifying  alternatives  is 
the  list  of  programs  identified  in  the  AFRL/VA  Strategic  Plan  and  the  Integrated  Concept 
Leaders’  roadmaps.  Since  Air  Vehicles  has  over  one  hundred  programs,  a  small  subset 
was  selected  for  analysis  by  the  Integrated  Concept  Leaders,  as  a  pilot  test  of  the 
methodology.  With  the  alternatives  identified,  scoring  of  each  alternative  was 
undertaken.  Technical  experts  in  Air  Vehicles  were  needed  to  complete  this  task. 

Finally,  decision  analysis  was  accomplished.  This  entails  taking  the  alternative 
scores  and  converting  each  score  to  a  value  score  using  single  dimension  value  functions. 
The  additive  value  model  was  implemented  to  identify  the  overall  fundamental  objective 
score  for  each  alternative.  Verification  of  mutual  preferential  independence  showed  the 
additive  value  model  was  valid.  A  rank  order  of  the  alternatives  was  developed  along 
with  graphs  showing  the  contribution  for  each  value  to  each  alternative.  Sensitivity 
analysis  was  used  to  determine  how  changes  in  the  weights  changed  the  decision  policy. 

Using  the  VA  cost  for  each  program,  the  benefit  to  the  warfighter  of  technology 
per  USAF  investment  cost  ranking  was  found.  The  overall  fundamental  objective  for  Air 
Vehicles  of  “Support  the  Warfighter  to  Achieve  Air  and  Space  Dominance”  was  agreed 
upon  by  the  Air  Vehicles’  Director  and  was  originally  identified  from  relevant  USAF 
doctrine  (summarized  in  Table  1  on  page  17).  Finally,  a  simple  mixed  integer  linear 
program  was  used  to  demonstrate  the  ability  to  optimize  an  investment  strategy  for  VA. 
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Automation  of  the  VA  value  model,  evaluation  measures,  alternative  scoring,  and 
graphing  of  results  was  implemented  using  Micrsoft  Excel.  The  method  builds  on 
Kirkwood’s  implementation  concepts  for  Excel  by  making  use  of  some  of  his  macros. 
Additional  macros  and  specialized  calculation  sheets  were  developed  to  simplify  the 
calculation  process. 

The  Search  for  the  Air  Vehicles’  Value  Hierarchy 

The  Method  for  Identifying  the  Hierarchy 

Development  of  the  Air  Vehicle  value  hierarchy  was  the  most  difficult  and 
important  part  of  this  effort.  The  question  was  what  is  important  to  Air  Vehicles?  Just 
asking  people  in  Air  Vehicles  what  is  important  to  them  would  not  necessarily  result  in  a 
value  model,  since  value  focused  thinking  is  a  new  concept  to  most  people  in  Air 
Vehicles.  While  decision-makers  have  always  used  their  “values”  to  make  decisions, 
they  have  not  always  explicitly  stated  these  values  and/or  quantified  them  in  a  systematic 
manner. 

In  the  literature,  two  methods  were  identified  for  developing  a  value  hierarchy. 
These  were  the  “Gold”  and  “Silver”  standards  (Parnell  et.  al.,  1998:  1338,1340).  The 
“Gold”  standard  is  the  simplest  method  since  it  would  use  existing  strategic  USAF 
literature.  However,  while  information  for  an  Air  Vehicles’  model  is  available,  no  single 
document  was  available  that  captures  all  of  Air  Vehicles’  values  concerning  technology. 
A  combined  standard  of  using  existing  literature  and  then  working  with  experts  in  Air 
Vehicles  to  fill  in  the  information  gaps  was  the  natural  way  to  progress  and  was  used  in 
this  analysis. 
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Results  from  the  Literature 


The  Air  Vehicles  mission  statement  states  they  should  “deliver  the  best  air  vehicle 
technologies  for  aerospace  dominance  against  all  threats  . . (AFRL/VA  Corporate 
Planning  Group,  1998:  5)  clearly  documenting  the  desire  to  support  the  warfighter. 
However,  reductions  in  funding  to  the  military  made  it  clear  that  cost  is  a  major 
constraint  for  providing  this  benefit.  Hence,  the  objective  of  this  study  is  to  select  Air 
Vehicles’  programs  that  maximize  support  to  the  warfighter  while  meeting  budget 
constraints. 

With  “Support  the  Warfighter”  identified  as  the  overall  objective,  the  next 
challenge  was  how  to  break  it  down,  since  it  is  not  directly  measurable.  The  question  that 
needs  to  be  answered  next  is  how  does  the  Air  Force  use  technology  to  support  the 
warfighter.  Again,  the  literature  was  consulted  to  better  understand  what  is  important. 

The  Air  Force  Strategic  Plan  states  the  vision  for  the  USAF  as  “Air  Force  people 
building  the  world’s  most  respected  Air  and  Space  Force  global  power  and  reach  for 
America”  (Air  Force  Strategic  Plan  Volume  2:  Performance  Plan,  Sep  1998:  i).  Key 
words  in  the  vision  statement  are  “global  reach”  and  “global  power.”  These  concepts  are 
recurring  themes,  which  also  appear  in  documents  such  as  New  World  Vistas,  Global 
Engagement,  and  Air  Force  2025.  Air  Force  2025  provided  a  good  starting  point  for 
defining  these  words  since  the  study  was  found  to  be  “very  valuable  to  the  study  director 
and  the  senior  leadership  of  the  Air  Force”  (Parnell  et.  al.,  1998:  1349).  It  was  publicly 
released  by  the  former  Chief  of  Staff  of  the  Air  Force,  General  Ronald  R.  Fogleman  and 
the  former  Secretary  of  the  Air  Force,  the  honorable  Sheila  E.  Widnall  in  1996. 
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Further  research  of  USAF  strategic  documents  shows  another  concept  that  is 
important  to  the  USAF.  Examination  of  New  World  Vistas,  Air  Force  Strategic  Plan  and 
Global  Engagement  identifies  information  superiority  as  one  of  the  core  competencies  of 
the  USAF.  Air  Force  2025  identifies  “awareness”  as  an  important  value  for  maintaining 
air  and  space  dominance.  Information  superiority  and  “awareness”  are  similar  concepts. 
Furthermore,  “awareness”  is  required  to  know  how  to  employ  global  power  and  to  know 
where  to  reach  out  against  an  adversary. 

The  values  “reach”,  “awareness”  and  “power”  cannot  be  measured  directly,  so 
these  values  also  need  to  be  broken  down.  For  each  of  the  top  three  values,  sub-values 
were  identified  as  “gain”  and  “sustain.”  Both  of  these  values  can  be  identified  from 
AFRUVA  Strategic  Plan  integrated  concepts.  Essentially,  Air  Vehicles  either  develops 
new  technologies  for  the  production  of  new  platforms  with  new  capabilities  (i.e.  gains 
new  capability  for  the  USAF)  or  improves  existing  platforms  (i.e.  sustains  the  existing 
Air  Force  fleet).  In  fact,  sustainment  of  the  USAF  fleet  is  one  of  Air  Vehicle’s  main 
efforts. 

Improvements  in  awareness,  reach  and  power  are  crucial  to  the  USAF  to  maintain 
air  and  space  dominance.  However,  according  to  1997  Defense  Technology  Area  Plan, 
technology  superiority  “has  been,  and  continues  to  be,  a  cornerstone  of  our  national 
military  strategy”  (The  Military  Critical  Technologies  List.  1997  Defense  Technology 
Area  Plan,  1997:  Introduction).  The  AFRL  mission  is 

To  Discover,  Develop,  Integrate,  and  Deliver  affordable  technologies  for 

Warfighting  Capabilities  ...  To  keep  our  Air  Force  the  best  in  the  world. 

(AFRL/VA  Corporate  Planning  Group,  1998:  4). 
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Air  Vehicles’  “Core  Values”  state  “the  first  essential  of  Aerospace  Power  is  pre¬ 
eminence  in  research”  (AFRL/VA  Corporate  Planning  Group,  1998:  6).  Joint  Vision 
2010  states: 

This  era  will  be  one  of  accelerating  technological  change.  Critical 
advances  will  have  enormous  impact  on  all  military  forces.  Successful 
adaptation  of  new  and  improved  technologies  may  provide  great  increases 
in  specific  capabilities.  Conversely,  failure  to  understand  and  adapt  could 
lead  today’s  militaries  into  premature  obsolescence  and  greatly  increase 
the  risks  that  such  forces  will  be  incapable  of  effective  operations  against 
forces  with  high  technology  (Joint  Vision  2010,  1996:  1 1). 

The  conclusion  is  “Technological  Superiority”  is  a  critical  value  for  Air  Vehicles. 

While,  “Technological  Superiority”  is  clearly  important,  the  strategic  documents 

examined  do  not  specifically  show  how  it  is  done.  According  to  van  Creveld, 

...  the  best  military  technology  is  not  that  which  is  “superior”  in  some 
absolute  sense.  Rather  it  is  that  which  ‘masks’  or  neutralizes  the  other 
side’s  strengths,  even  as  it  exploits  his  weaknesses”  (van  Creveld,  1989: 

320). 

This  conclusion  by  van  Creveld  implies  that  technology  does  not  dominate  an  enemy 
because  of  its  absolute  attributes.  A  technology  must  be  compared  with  an  adversary’s 
technology  to  determine  whether  or  not  it  dominates.  With  this  information,  an 
understanding  of  “technological  superiority”  as  it  pertains  to  military  operations  becomes 
possible.  Further  information  for  breaking  down  “technological  superiority”  can  be 
found  in  Appendix  A:  USAF  Strategic  Documents  Literature  Search  Results. 

The  Air  Vehicles  Hierarchy 

With  candidate  values  in  hand,  meetings  to  develop  and  refine  the  Air  Vehicles 
Value  hierarchy  began  in  December  1998.  Detailed  documentation  of  the  meetings  can 
be  found  in  Appendix  B:  Documentation  of  Meetings  with  VA.  Figure  2,  Figure  3,  and 
Figure  4  show  the  Air  Vehicles  Value  model  with  definitions  for  each  value. 
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Figure  2:  Top  Level  Value  Model  for  Air  Vehicles 

Figure  2  shows  the  overall  fundamental  objective  and  the  most  important  benefit 
Air  Vehicles  adds  to  the  Air  Force.  Air  Vehicles  supports  the  warfighter  in  achieving  and 
maintaining  aerospace  dominance.  However,  there  is  an  investment  cost  associated  with 
Air  Vehicles  providing  this  benefit.  Since  the  benefit  Air  Vehicles  provides  to  the 
warfighter  cannot  be  directly  measured,  the  overall  value  must  be  broken  down,  until  the 
lower  level  supporting  concepts  are  reached  that  can  be  measured.  The  values  “reach,” 
“awareness,”  “power,”  and  “technological  superiority”  were  accepted  by  the  Air  Vehicles 
Directorate  as  sufficiently  descriptive  of  what  is  important  about  supporting  the 
warfighter.  Figure  2  shows  the  definitions  for  the  values,  which  have  been  improved  by 


Air  Vehicles,  but  can  be  traced  back  to  Air  Force  Strategic  doctrine. 
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Figure  3:  Value  Model  Breakdown  of  Reach,  Awareness  and  Power 


Technological 

Superiority 


VA’s  “Savings  Multiple  for  Cost”  measure  in 
widespread  use.  Defined  as  the  ratio  of  cost 
savings  potential  to  the  U.S.  Air  Force  to  the  Air 
Force  technology  investment  cost. 


Afford 

’  Be  Efficient 

i . ►[ 

qual  or  better  capability  to  achieve  the  warfighter  mission. 


Practices  to  reduce  the  effort  to  develop  technology. 


Revolutionary  technology  advances  that  increase  technological  options 
for  extending  or  increasing  the  spectrum  of  warfighting  capabilities. 


Programmatic  risk  is  the  likelihood  of  not  completing  the  proposed 
program  within  the  proposed  time  for  the  proposed  cost. 


Technology  that  makes  military  operations  easier  to  implement. 


Technology  advances  that  render  an  adversaries  military  capability 
ineffective  or  bypasses  an  adversary’s  strength. 


Figure  4:  Value  Model  Breakdown  of  Technological  Superiority 

Technological  superiority  cannot  be  directly  measured,  either.  It  was  therefore, 


further  broken  down,  as  seen  in  Figure  4.  Considerable  rearrangement  took  place  on  the 
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values  and  on  what  was  later  to  become  evaluation  measures.  The  definitions  and  values 
were  modified  to  their  present  form  from  the  inputs  of  the  Air  Vehicles  Directorate. 


Figure  5:  The  Air  Vehicles’  Value  Hierarchy 


Technically,  a  value  model  should  not  be  broken  into  a  single  value  (Keeney, 
1992:  78).  However,  this  value  model  is  also  representing  what  is  important  to  the  AFRL 
at  its  higher  levels,  while  it  becomes  specific  to  Air  Vehicles  at  lower  levels.  Figure  5 
shows  the  Air  Vehicles’  value  model.  The  highest  tiers  of  “Support  the  Warfighter”  and 
“Reach,  Awareness,  Power,  and  Technological  Superiority”  are  directly  applicable  to 
AFRL.  The  values  one  tier  below  “Reach,  Awareness,  Power  and  Technological 
Superiority”  are  specific  to  the  ten  Directorates  of  AFRL,  but  are  not  collectively 
exhaustive,  except  for  the  Air  Vehicles’  Directorate.  For  AFRL  purposes,  the  “Cost  to 
Air  Vehicles  Directorate”  would  be  changed  to  “Cost  to  AFRL.” 
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Developing  the  Air  Vehicles’  Value  Model 

Measuring  Change  in  Support  to  the  Warfighter  due  to  Technology 

Development  of  evaluation  measures  was  the  next  step  in  the  process.  Early  on,  it 
was  realized  the  evaluation  measures  would  need  to  directly  or  indirectly  measure  the 
technological  change  in  warfighting  capability  rather  than  a  specific  capability  level 
brought  about  by  the  technology.  Alexander  and  Mitchel  developed  a  model  for 
measuring  technological  change  and  in  one  version  of  their  model,  they  allow  vectors  of 
“product”  and  “use  variables”  to  change  with  time  while  holding  resource  input  vectors 
constant  (Alexander  and  Mitchel,  1985:  167-168). 

The  concept  of  measuring  the  difference  between  current  state  of  the  art  and 
future  state  of  the  art  was  adopted  for  the  Air  Vehicles  value  model.  However, 
measuring  technological  change  for  the  warfighter  was  not  directly  measured  with  the 
value  model.  The  model  actually  measures  the  “change  in  warfighter  support  due  to 
technology,  which  is  a  function  of  the  technological  change  brought  about  by  Air 
Vehicles. 

The  change  in  warfighter  support  due  to  technology  can  be  determined  between 
an  existing  system  (reference  system)  without  the  technology  and  a  system  with  the 
proposed  technology.  The  system  without  the  technology  is  the  assumed  current 
capability  while  a  system  with  the  technology  is  assumed  available  at  a  specified  date  in 
the  future.  A  technology  program  represents  a  potential  change  from  a  current  reference 
system  to  a  future  new  system.  The  difference  in  value  between  the  new  system  and  the 
old  system  quantifies  the  change  in  warfighter  technological  support  for  that  evaluation 
measure.  The  deployed  state  of  the  art  for  an  evaluation  measure  will  be  defined  as  the 
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highest  scoring  known  reference  system  for  the  objective.  The  word  “deployed”  is  used 
because  state  of  the  art  in  technology  may  not  be  synonymous  with  the  technology  actual 
found  in  the  USAF. 

It  is  possible  that  a  reference  baseline  system  does  not  exist.  This  can  happen 
when  a  revolutionary  weapon  system  is  planned  for  development.  To  use  the  Air 
Vehicles’  value  model,  however,  a  reference  system  is  required.  In  this  situation,  each 
evaluation  measure  should  be  examined  separately  to  determine  if  any  existing  system  is 
similar  in  that  particular  attribute. 

Several  existing  systems  may  have  to  be  used  to  define  the  deployed  state  of  the 
art  for  each  attribute.  For  example,  Air  Vehicles  is  working  on  an  unmanned  fully 
resuseable  space  vehicle.  The  only  known  example  of  a  reuseable  space  vehicle  in 
existence  today  is  the  Space  Shuttle.  However,  the  Space  Shuttle  is  a  manned  system, 
and  has  a  much  higher  payload-to-orbit  capability  (due  to  size  and  scale  effects)  than  the 
proposed  reuseable  space  vehicle.  Comparison  of  the  two  vehicles  would  require  scaling 
payload  capacity  and  would  have  to  subtract  hardware  preparation  time  that  is  specific  to 
manned  space  flight.  It  is  better  to  compare  the  new  unmanned  resuseable  vehicle  with 
existing  expendable  launch  vehicles  to  define  a  reference  deployed  state  of  the  art 
capability  in  technology  (see  Appendix  H:  Air  Vehicles’  Programs  and  Scoring  for 
details).  If  no  reference  system  can  be  found  for  comparison,  the  evaluation  measure 
should  achieve  its  maximum  value. 

Trading  off  Attributes  in  Favor  of  Others 

A  problem  that  arises  with  measuring  the  change  warfighter  technological  support 
brings  is  that  negative  value  can  result.  Intuitively,  this  happens  when  the  new  system 
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has  lower  value  then  the  reference  system  or  the  deployed  state  of  the  art  for  a  particular 
evaluation  measure.  For  example,  it  may  be  desirable  to  lower  horsepower  in  an 
automobile  to  allow  improvement  in  the  fuel  economy.  If  just  a  single  evaluation 
measure  was  required  to  score  a  program  (i.e.  only  horsepower  mattered  to  the  decision¬ 
maker),  and  it  reduced  capability  from  the  current  deployed  state  of  the  art,  that 
technology  would  not  be  developed.  Because  the  model  is  a  multiattribute  model, 
tradeoffs  exist  between  the  evaluation  measures  and  the  values  in  the  model.  It  is 
possible  one  evaluation  measure  will  be  penalized  in  order  to  gain  a  much  larger 
advantage  in  another  evaluation  measure. 

While  there  is  no  theoretical  reason  preventing  the  use  of  negative  value,  value 
focused  thinking  was  developed  with  the  assumption  that  value  is  always  positive. 
Evaluation  measures  that  would  intuitively  have  resulted  in  negative  value  actually 
scored  positively.  There  is  no  problem,  if  the  value  function  for  each  evaluation  measure 
is  calibrated  for  the  “deployed  state  of  the  art.”  When  a  program  scores  below  the 
deployed  state  of  the  art  value,  it  can  be  understood  that  the  program  is  decreasing 
capability  for  the  warfighter.  When  a  program  scores  above  the  deployed  state  of  the  art 
value,  it  is  understood  that  the  program  is  increasing  capability  for  the  warfighter. 

As  reviewed  in  Chapter  2,  value  functions  can  be  measured  on  interval  scale 
(Luce,  Bush  and  Gallanter,  1963:  15;  von  Winterfeldt  and  Edwards,  1986:  209-210). 
Using  the  value  function  for  an  evaluation  measure,  it  is  possible  to  calculate  the 
difference  in  value  for  a  program  and  the  deployed  state  of  the  art.  Because  the  value  for 
the  program  and  the  deployed  state  of  the  art  are  measured  on  the  same  interval  scale,  the 
difference  between  the  two  will  be  measured  on  a  ratio  scale.  This  difference  is  defined 
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as  the  change  in  warfighter  technological  support  where  a  value  of  zero  represents  the 
current  deployed  state  of  the  art  for  that  evaluation  measure. 

This  result  has  three  implications.  First,  change  in  warfighter  technological 
support  is  being  measured  on  a  scale  with  an  absolute  zero.  This  means  the  current 
deployed  state  of  the  art  has  been  estimated  and  quantified  (within  the  accuracy  of  the 
evaluation  measures  and  scoring  on  the  programs).  Second,  it  is  possible  to  directly 
compare  one  program  to  another  in  how  it  impacts  change  in  warfighter  technological 
support  (see  (Roberts,  1979:  71-74  for  a  discussion  on  meaningful  statements  with 
ratios).  Third,  defining  change  in  warfighter  technological  support  on  a  ratio  scale  means 
the  mathematical  programming  assumption  of  proportionality  (Winston,  1994:  53-54)  can 
be  met. 

The  evaluation  measures  were  developed  with  the  aid  of  specially  designed  Excel 
macros  and  templates.  The  macros  and  the  templates  are  documented  in  Appendix  C: 
Evaluation  Measure  Templates  and  Macros.  The  process  of  developing  evaluation 
measures  follows  the  methods  describe  in  Chapter  2. 

Elicitation  of  Single  Dimension  Value  Functions 

There  are  infinitely  many  possible  shapes  for  an  evaluation  measure.  In  all  cases, 
the  ranges  and  shapes  of  all  of  the  evaluation  functions  were  developed  with  the  help  of 
Air  Vehicles  leaders  and  technical  experts.  The  types  of  evaluation  measures  used  were 
discrete,  exponential,  piecewise  linear  or  combinations  of  exponential  functions 
(piecewise  exponential). 
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An  evaluation  measure  with  categories  is  a  discrete  measure  with  a  discrete 
function.  A  special  case  of  a  discrete  function  is  a  binary  function.  In  this  situation,  an 
alternative  either  receives  all  of  the  value  or  none  of  the  value  for  the  function. 

Determination  of  whether  a  value  function  should  be  an  exponential  function,  a 
piecewise  linear  function,  or  some  other  function  required  input  from  the  Air  Vehicles’ 
experts.  For  this  project,  an  exponential  function  was  the  first  choice  function.  This  is 
because  of  its  flexibility  to  model  diminishing  returns  on  value  for  convex  curves  or  the 
ability  to  show  only  marginal  gains  in  value  until  maximum  value  for  concave  curves  is 
achieve. 

Questions  were  asked  to  gain  an  understanding  of  the  shape  of  the  curve.  If  the 
exponential  curve  was  clearly  not  correct,  a  piecewise  linear  curve  was  tried.  If 
additional  questions  showed  a  location  in  the  value  function  of  large  increase  or  decrease 
followed  by  little  change  in  the  value,  an  S-curve  was  considered  (see  Appendix  D:  S- 
Curves  as  a  Value  Measure  for  details).  If  the  resulting  function  was  an  asymmetrical  S- 
curve,  multiple  exponential  curves  were  combined  to  develop  the  value  function. 

An  example  of  an  evaluation  measure  can  be  found  in  Figure  6.  “Availability”  is 
an  evaluation  measure  under  “Reach”  under  “Gain.”  The  discussion  for  this  evaluation 
measure  with  Air  Vehicles  first  considered  the  definition.  The  definition  needed  to 
include  “a  single  system  which  prevents  the  aircraft  from  performing  its  mission”  in  the 
definition.  Modern  aircraft  are  typically  available  60%-65%  of  the  time.  The  measure  is 
Pan /Pa o  >  which  is  the  ratio  of  the  probability  of  being  available  for  the  new  system 
versus  the  probability  of  being  available  for  the  old  system. 
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The  range  of  the  ratio  was  set  to  never  go  higher  than  1.5,  which  is  an  increase  in 
availability  for  the  new  system.  A  value  of  0  is  the  worst  the  new  system  can  be 
available,  which  is  not  available  at  all.  This  allows  for  an  increase  up  to  100%  in  most 
situations  and  for  decreasing  availability  in  the  case  of  a  new  weapon  system  performing 
a  new  mission.  The  measure  assumes  time  between  failure  and  time  to  repair  is 
considered  in  determining  the  probability  of  availability.  This  evaluation  measure  is 
mutually  exclusive  with  Regeneration  because  it  does  not  cover  repair  of  battle  damage 
or  the  time  to  replenish  consumeables  on  the  weapon  system. 

The  evaluation  measure  function  shown  in  Figure  6  describes  the  function  from  a 
Pan /Pa o  from  0  to  1.  The  equation  that  describes  PAN/PA0  from  1  to  1.5  is  shown  in 


Figure  7.  The  only  difference  between  Figure  6  and  Figure  7  is  that  “Sustain 
Availability”  will  never  be  decreased  by  adding  on  new  technology,  while  “Gain 
Availability”  has  the  potential  of  decreasing  in  favor  of  other  attributes. 


Evaluation  Measure:  Gain  Availability 

Range  Units:  Unitless 
Definition:  Pn/Po  where: 

Pn  is  the  probability  the  weapon  system  is  availabile  with  the 
proposed  new  technology 

Po  is  the  probability  the  weapon  system  is  available  without  the 
proposed  technology. 

Availability  is  the  probability  the  weapon  system  can  complete  its 
normal  combat  mission  on  demand.  A  weapon  system  is  not 
available  when  mission  critical  equipment  fails. 

Comments:  This  measure  combines  both  the  time  between 
failures  or  the  reliability  of  the  system  with  the  time  to  repair  the 
system.  This  alternative  results  in  no-change  when  Pn-Po. 


V(X)  =  A-B*exp(-X/R) 
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Figure  6:  Gain  Availability  Evaluation  Measure 
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V(X)  =  A-B*exp(OC/R) 


Evaluation  Measure:  Sustain  Availability 

Range  Units:  Unitless 
Definition:  Pn/Po  where: 

Pn  is  the  probability  the  weapon  system  is  availabile  with  the  proposed 
new  technology 

Po  is  the  probability  the  weapon  system  is  available  without  the  proposed 
technology. 

Availability  is  the  probability  the  weapon  system  can  complete  its  normal 
combat  mission  on  demand.  A  weapon  system  is  not  available  when 
mission  critical  equipment  fails. 

Comments:  This  measure  combines  both  the  time  between  failures  or 
the  reliability  of  the  system  with  the  time  to  repair  the  system.  The 
availability  of  a  weapon  system  will  never  be  intentionally  decreased. 

This  alternative  results  in  no-change  when  Pn=Po. 


B  |  R  |  X  |  Value  1  Value  Function  Range 


36785.25  0.15238  1.00  i  50.0 
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Figure  7:  Sustain  Availability  Evaluation  Measure 

An  example  of  a  constructed  evaluation  measure  can  be  seen  in  Figure  8.  This 


evaluation  measure  was  developed  to  show  the  importance  of  how  outside  customers  rate 
Air  Vehicles’  programs.  Making  the  “Top  10  ATD”  or  top  10  advanced  technology 
development  list  provides  the  most  value.  An  “endorsement”  from  a  System  Program 


Office  or  a  Major  Command  is  also  valuable,  but  is  far  less  important  than  being  in  “Top 


10  ATD.”  This  change  in  value  between  the  two  shows  the  tremendous  importance  of 


being  in  the  “Top  10  ATD”  group. 


ReScale 


Range  Units:  Categories  as  defined 

Definition:  The  rating  of  the  program  given  by  the  appropriate 

agency. 

Comments:  A  program  in  the  Top  10  ATD's  has  tremendous 
importance.  Endorsement  by  a  SPO  or  MAJCOM  is  good,  but  far 
less  useful. 


Evaluation  Measure: 


Be  Efficient:  User  Rating 


Abreviation 

XI _ 

X2 _ 

X3 _ 


Category 
No  Rating 
Endorsed 
Top  10  ATD 


Value 


Category  Definition 


_ 0  Not  rated. _ 

20  Written  endorsement  by  MAJCOM  or  SPO. _ 

100  In  the  MAJCOM's  top  10  Advanced  Technology  Development  List 


Figure  8:  User  Rating  under  Technological  Superiority  under  Be  Efficient 
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Appendix  F:  Air  Vehicles’  Evaluation  Measures,  shows  all  of  the  evaluation 
measures  used  in  the  Air  Vehicles  value  model.  Definitions  for  each  evaluation  measure 
are  provided  along  with  comments,  mathematical  representations  of  the  functions  and 
graphical  representations  of  the  functions.  Figure  9  shows  the  entire  Air  Vehicles  value 
model. 
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Figure  9:  Air  Vehicles  Value  Model 


Using  the  Additive  Value  Function 

From  the  literature,  it  was  determined  that  a  majority  of  small  value  hierarchies 
under  certainty  can  use  the  simple  additive  value  function  for  combining  multiple  single 
dimension  value  functions  into  an  overall  figure  of  merit  with  no  loss  in  accuracy 
(Clemen,  1996:  579;  Fischer,  1977:  299).  However,  in  a  minority  of  cases,  the  additive 
value  function  cannot  be  used  because  mutual  preferential  independence  does  not  hold. 
To  be  certain  mutual  preferential  independence  holds,  a  test  was  conducted  with  Air 
Vehicles.  The  meetings  were  held  with  the  Integrated  Concept  Leaders  and  the  Strategic 
Integrated  Product  Team  Leader. 

A  matrix  with  every  evaluation  measure  in  the  value  model  in  the  columns  and 
every  evaluation  measure  in  the  rows  was  developed.  The  measures  in  the  rows  were  set 
at  three  arbitrary  low,  medium  and  high  levels.  Each  evaluation  measure  in  the  columns 
was  incremented  from  an  arbitrary  low  level  to  an  arbitrary  high  level.  The  arbitrary 
levels  were  chosen  not  to  be  at  the  lowest  or  highest  point  of  the  evaluation  measure.  Air 
Vehicles  was  asked  to  compare  each  measurement  in  each  row  at  a  fixed  value  level  with 
each  evaluation  measure  in  the  columns.  The  purpose  was  to  determine  if  they  would 
always  prefer  the  highest  level  to  the  lowest  level  for  each  evaluation  measure,  regardless 
of  how  the  fixed  measure  was  set. 

The  team  reported  they  would  not  change  their  preferences  for  any  of  the  other 
evaluation  measures  based  on  any  particular  measure  being  fixed  at  a  particular  level. 
This  result  combined  with  the  literature  search  result  that  given  mutual  preferential 
independence,  additive  value  functions  give  up  very  little  accuracy,  supported  the 
additive  value  function  choice  for  the  Air  Vehicles’  value  model. 
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Weighting  of  Values  using  a  Hypothetical  Example 

The  method  of  swing  weights  (Kirkwood,  1997:  68-70)  described  in  Chapter  2: 
Assessing  Weights  on  page  29  was  modified  for  this  thesis.  The  major  problem  with  the 
method,  outlined  from  the  literature,  is  the  requirement  for  a  decision-maker  to  consider 
the  ranges  of  all  of  the  values  and  evaluations  measures  under  the  two  values  being 
compared  (Keeney  and  Raifa,  1976:  125).  The  value  model  for  Air  Vehicles  is  complex 
and  contains  31  evaluation  measures.  It  is  nearly  impossible  to  consider  the  ranges  for  5- 
10  evaluation  measures  simultaneously  while  assessing  how  much  more  important  one 
value  is  to  another. 

A  different  process  was  employed.  The  concept  is  to  ask  how  much  value  is 
gained  “swinging”  from  the  lowest  to  highest  in  one  evaluation  measure  compared  to 
“swinging”  from  the  lowest  to  highest  in  another  evaluation  measure  across  two  different 
value  model  columns.  When  a  decision-maker  provides  a  ratio  of  the  importance,  the 
relationship  between  the  highest  value  above  each  evaluation  measure  will  be  established. 

To  understand  how  this  happens,  the  concept  of  local  weights  and  global  weights 
must  be  discussed.  A  method  for  defining  local  weights  was  defined  in  Chapter  2. 
Essentially,  a  group  of  values  or  evaluation  measures  is  considered  under  a  single  higher 
level  value.  The  decision-maker  is  asked  to  rank  order  the  values/evaluation  measures 
and  then  is  asked  questions  requiring  him  to  compare  two  at  a  time.  The  decision-maker 
will  be  asked  as  many  comparison  questions  as  there  are  values/evaluation  measures  at 
that  tier  minus  one.  The  last  equation  is  that  all  weights  must  sum  to  one.  What  results  is 
a  set  of  simultaneous  equations  that  are  easily  solved. 
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Global  weights  combine  all  of  the  local  weights  at  the  highest  tiers  of  the  value 
model  down  to  the  lowest  tier  to  develop  an  overall  weight  for  each  evaluation  measure. 
Figure  10  shows  a  sample  value  model  with  local  weights  determined  from  a  hypothetical 
decision-maker.  The  global  weight  for  Ml  was  found  by  multiplying  all  of  the  local 
weights  at  higher  tier  levels  by  the  local  weight  of  Ml.  That  is,  the  global  weight  for  Ml 
is  (0.25)(0.4)(1)=  0.1.  The  value  “VT”  has  a  weight  of  one,  since  it  has  no  other  value  at 
the  same  tier  level. 


Figure  10:  Difference  between  Global  and  Local  Weights 


Similar  to  the  swing  weight  method,  the  decision-maker  is  asked  to  rank  order  the 
evaluation  measures  before  any  information  is  elicited.  The  ability  for  weights  to  change 
based  on  the  evaluation  measure  range  is  still  a  factor  in  this  method.  In  Figure  10,  M2 
would  be  more  important  than  Ml  and  M4  would  be  more  important  than  M3.  Next,  the 
decision-maker  is  asked  to  compare  two  evaluation  measures  (considering  the  ranges 
they  cover)  between  two  values.  For  example,  the  decision-maker  could  be  asked,  “How 
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much  more  important  is  M2  to  M3?”  In  this  case,  the  decision-maker  would  have 
answered  IVz  times  as  important.  The  relationship  refers  to  the  global  weights  of  M2  and 
M3. 


Next,  the  standard  method  of  swing  weights  is  employed  to  find  the  local  weights 
for  Ml,  M2,  M3,  and  M4.  For  this  example,  it  will  be  assumed  the  decision-maker 
believes  M2  to  be  three  times  more  important  than  Ml  and  M4  to  be  twice  as  important 
as  M3.  The  equations  for  finding  the  local  weights  are: 


WM2 

Wm 


=  3Wm, 
+WM2  =  l 
+  Ww4=l 


Equation  3 


with  the  an  equal  number  of  unknowns  and  equations,  the  local  weights  can  be  easily 


solved  to  find  Wmi  =  0.25,  Wm2  =  0.75,  Wm3  =  0.33,  and  Wm4  =  0.67.  The  variable  “W” 


represents  the  weight  for  the  subscripted  evaluation  measure  or  value  as  shown  in  Figure 


10. 


Next,  the  decision-maker  is  asked  to  compare  evaluation  measure  M2  with  M3  to 
set  the  relationship  between  VI  and  V2.  The  result  is  the  decision-maker  never  has  to 
consider  more  than  two  evaluation  measures  for  every  value  model  relationship  he  is 
asked  to  find.  It  will  be  assumed  the  decision-maker  believes  M2  to  be  1 .5  times  more 
important  than  M3.  This  relationship  does  not  compare  local  weights  like  was  done  with 
Equation  3,  but  compares  global  weights.  Therefore,  a  slightly  different  set  of  equations 
must  be  solved  for  as  follows: 


WutW„=1.5(Wu,Wvl) 

WVI+WV2=\  Equation  4 
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The  first  equation  defines  the  relationship  “elicited”  from  the  decision-maker  in  terms  of 
local  weights.  The  second  equation  is  the  standard  requirement  that  all  local  weights 
must  sum  to  one.  Solving  the  simultaneous  equation  using  the  answers  from  Equation  3 
results  in  Wvi  =  0.4  and  Wv2  =  0.6. 

In  the  “elicited  relationship  for  measures  between  values”  equation,  the  weight  for 
value  “VT”  Figure  10  does  not  appear  in  the  equation.  Although  the  weight  for  this  value 
is  one,  it  would  not  have  mattered  if  it  had  been  any  other  positive  number  less  than  one 
(zero  renders  calculation  of  weight  irrelevant).  This  is  because  both  VI  and  V2  are 
subvalues  to  VT.  When  comparing  the  global  weights  between  V 1  and  V2  using 
evaluation  measures  M2  and  M3,  the  weight  for  VT  would  appear  on  both  sides  of  the 
equation.  This  value’s  weight  and  any  value  above  VT  can  be  “ignored”  since  it  will 
always  cancel  out  of  the  equation  relating  the  two  global  weights. 

Air  Vehicles’  Weights 

For  Air  Vehicles,  the  weights  were  elicited  from  both  the  Directorate  Senior 
Leadership  and  from  the  Integrated  Concept  Leaders.  The  Directorate  Senior  Leadership 
provided  weights  for  the  values,  using  the  new  method  discussed  in  the  previous  section 
while  the  swing  weights  method  was  used  to  query  the  Integrated  Concept  Leaders. 

Table  6  documents  the  weights  elicited  from  Air  Vehicles  while  Appendix  G:  Air 
Vehicles’  Weight  Calculations  shows  the  details  for  the  calculations. 

The  Directorate  Senior  Leadership  was  asked  several  questions  in  an  attempt  to 
gain  consistent  answers.  The  senior  leader  was  first  asked  to  rank  order  the  evaluation 
measures  to  determine  the  most  important  measure  for  each  value.  Then,  only  the  most 
important  measures  were  compared.  This  is  important  because  it  allowed  the  decision- 
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maker  to  compare  something  he  was  most  familiar  with  and  deals  with  often.  Another 
concern  would  be  to  not  compare  the  highest  ranking  evaluation  measure  with  the  lowest 
level  in  another  value  since  this  would  inflate  the  decision-maker’s  answers.  Careful 
attention  was  paid  to  explaining  what  the  ranges  of  the  evaluation  measures  being 
compared  were.  Besides  changing  the  ultimate  weights  in  the  model,  it  was  important  the 
decision-makers  were  considering  the  same  evaluation  measure  as  was  documented  in  the 
value  model. 
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for  the  Air  Vehicles  Value  Model 
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Modeling  Assumptions 

Specific  to  R&D  and  Air  Vehicles 

Biondi  and  Galli  stated  assumptions  that  allow  technology  to  be  quantified.  Their 
first  premise  that  development  of  technology  is  not  completely  random  is  critical  here.  If 
development  of  technology  is  random,  than  all  attempts  to  model  technology  are  doomed. 
Another  premise  they  propose  is  that  technology  can  be  quantified  and  that  it  is  possible 
to  identify  “higher-quality  configurations.”  Again,  this  premise  is  fundamental  to  this 
work. 

The  difference  in  current  technology  or  deployed  state  of  the  art  is  compared  with 
the  potential  deployed  state  of  the  art  in  many  of  the  evaluation  measures.  It  is  assumed 
that  the  alternatives  chosen  to  represent  the  current  technological  deployed  state  of  the  art 
are  in  fact  the  deployed  state  of  the  art.  Failure  to  adhere  to  this  assumption  will  inflate 
the  value  of  new  technology,  which  will  cause  erroneous  results. 

The  programs  or  alternatives  are  assumed  to  represent  a  typical  sample  of 
programs  in  Air  Vehicles.  Seven  programs  were  analyzed  in  this  effort.  Air  Vehicles  has 
stated  there  are  100-200  programs  in  various  stages  of  work.  The  Integrated  Concept 
Leaders  at  Air  Vehicles  were  specifically  asked  to  identify  programs  they  thought  would 
represent  Air  Vehicles’  program  portfolio  that  would  be  diverse  enough  to  test  the  model. 

Levels  of  technical  risk  are  assumed  to  correlate  with  R&D  categories  of  basic 
research,  applied  research,  and  development  research.  In  order  of  highest  risk  to  lowest 
risk,  basic  research  ranks  highest,  applied  research  is  lower,  and  development  research  is 
lowest.  As  such,  technical  risk  is  predefined  and  assumed  constant  for  any  program  in  a 
particular  category.  Technical  risk  is  held  constant  when  programmatic  risk  is  varied. 
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Specific  to  the  Value  Model  and  the  Resource  Allocation  Models 

Many  raw  scores  of  attributes  were  measured  using  ratios,  percentage  change,  or  a 
simple  difference  between  deployed  state  of  the  art  and  the  new  proposed  deployed  state 
of  the  art.  This  aggregation  allows  mission  dependent  information  to  be  cancelled  from 
the  measurement  and  results  in  a  manageable  number  of  evaluation  measures  in  the  value 
model. 

The  programs  or  alternatives  in  the  initial  study  are  assumed  to  not  interact.  For 
example,  the  value  of  Program  A  and  Program  B  might  together  have  higher  or  lower 
value  than  the  sum  of  the  values  of  the  two  programs.  One  method  to  address  this  would 
be  to  evaluate  a  new  program  consisting  of  the  two  programs  combined.  This  method  is 
imperfect  at  best,  and  further  work  is  needed  to  relax  the  assumption. 

In  reality,  Air  Vehicles  does  not  fund  their  programs  all  at  once  and  up  front. 

Since  total  funding  requirements  for  Air  Vehicles’  programs  were  provided,  it  is  assumed 
these  numbers  represented  the  present  value  of  all  future  yearly  funding  required  to 
complete  the  program.  Yearly  cash  flow  constraints  are  easily  added  to  the  mathematical 
optimization  programs  used  in  this  analysis. 

Yearly  funding  profiles  for  the  programs  scored  were  not  available  at  the  time 
analysis  was  being  completed.  It  is  assumed  that  the  programs  are  funded  as  a  lump  sum 
dollar  amount.  Clearly,  this  is  not  true  in  reality,  and  the  models  being  used  could  be 
improved  to  consider  periodic  funding  for  the  programs. 

Any  alternative  that  scores  beyond  the  highest  value  for  an  evaluation  measure 
receives  the  maximum  value  of  100  for  that  measure.  Similarly,  any  alternative  that 
scores  below  the  lowest  value  for  an  evaluation  measure  receives  the  lowest  value  of  zero 
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for  that  evaluation  measure.  This  prevents  an  alternative  from  being  misrepresented  in 
the  value  model  and  prevents  biasing  of  the  results.  If  the  decision-maker  ever  feels  an 
alternative  should  score  outside  of  these  boundaries,  the  evaluation  measure  should  be 
rescaled  accordingly. 

Using  the  Air  Vehicles  Value  Model 

Alternatives 

The  starting  point  for  identification  of  alternatives  or  technology  programs 
is  the  Air  Vehicle’s  Strategic  plan  and  the  Integrated  Concept  Leaders’  roadmaps,  which 
list  programs  under  each  integrated  concept.  From  these  technologies,  Air  Vehicles 
provided  a  subset  of  programs  for  analysis  (see  Figure  7).  The  programs  were  chosen  so 
all  higher  tier  values  would  score  (thereby  testing  the  model)  and  so  each  of  Air 
Vehicles’  three  Integrated  Concepts  would  have  programs  scored.  The  “Scoring  Areas” 
show  values  where  each  program  contributed.  All  programs  except  “Composite  Repair 
of  Aircraft”  scored  under  “Reach”  under  “Gain”.  Three  programs  contributed  under 
“Reach”  under  “Sustain,”  one  program  contributed  under  “Awareness,”  and  four 
programs  brought  improvement  to  “Power.”  All  scoring  information  came  from  the  Air 
Vehicles’  program  managers  or  someone  they  designated  as  knowledgeable  about  their 
program. 

All  of  the  scoring  functions  were  implemented  in  Excel.  Air  Vehicles  also 
provided  their  investment  costs  required  to  perform  each  program.  Elicited  scores  and 
descriptions  of  each  program  can  be  found  in  Appendix  H:  Air  Vehicles’  Programs  and 
Scoring. 
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Alternative/Information 

Program  || 

HSE 

22 

Reach/Gain 

Reach/Sustain 

Awareness 

Power 

UAV 

Sustainment 

Space 

Weapons  Bay  Noise  Suppression 

D 

Q 

B 

B 

B 

Composite  Repair  of  Aircraft 
Strctures 

B 

■ 

X 

■ 

■ 

■ 

X 

■ 

ITAC 

B 

B 

B 

B 

SITE-M 

□ 

B 

B 

B 

Actuation  Health-Monitoring 

Systems  (HMS) 

E 

X 

X 

X 

■ 

X 

Space-Based  System-of-systems 
Advanced  Dv  Demo  &  Validation 

F 

X 

■ 

X 

mTv 

y 

B 

5 

5 

B 

B 

5 

5J 

Table  7:  Programs  Scored  Using  the  Air  Vehicles’  Value  Model 


Besides  identifying  technology  programs,  it  was  also  necessary  to  identify 
baseline  warfighter  scores  or  the  current  deployed  state  of  the  art  for  the  proposed 
technology  program.  The  purpose  of  this  was  to  allow  identification  of  the  change  from 
the  current  deployed  state  of  the  art  in  technology  to  the  new  proposed  deployed  state  of 
the  art.  Scoring  the  change  in  technology  allowed  quantification  of  the  value  of 
technology. 

A  hypothetical  “no  change”  alternative  was  determined.  This  alternative  was  the 
one  that  results  in  no  improvement  in  value  between  an  existing  reference  system  and  the 
new  technology  system.  The  no  change  alternative  provides  insight  to  the  decision¬ 
maker  about  whether  or  not  a  program  actually  improves  the  warfighter’s  capability  and 
whether  or  not  it  should  be  pursued.  It  also  represents  the  deployed  state  of  the  art.  At 
the  highest  level  of  the  hierarchy,  it  shows  the  deployed  state  of  the  art  for  the  warfighter 
as  defined  by  Air  Vehicles. 
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Program  Evaluation 

Program  evaluation  is  the  process  of  taking  the  raw  scores  for  each  program  and 
converting  them  to  an  overall  value  using  the  value  model.  The  process  is  to  convert  the 
raw  scores  provided  by  Air  Vehicles  into  value  using  the  single  dimensional  value 
functions  for  each  attribute.  Using  an  additive  value  function,  the  value  functions  were 
“rolled  up”  into  a  single  measure  of  benefit  to  the  warfighter.  This  process  was 
completed  for  every  alternative  under  consideration.  The  alternatives  were  then  ranked 
according  to  their  overall  value.  Graphs  were  developed  showing  how  each  alternative 
contributes  in  different  areas  to  the  overall  value. 

Because  Air  Vehicles  supplied  the  expected  cost  for  each  alternative,  the  benefit 
to  cost  ratio  was  calculated  for  each  alternative.  The  ratio  allows  determination  of  the 
largest  “bang  for  the  buck.”  The  result  was  a  prioritized  list  identifying  the  alternatives 
that  provide  the  most  benefit  per  cost.  It  represents  the  first  cut  at  developing  a  portfolio 
of  programs  that  optimize  the  benefit  of  the  programs  while  meeting  budgetary 
constraints.  However,  the  portfolio  is  not  guaranteed  to  be  optimal.  Therefore,  a  simple 
mixed  integer  program  was  developed  to  maximize  the  benefit  to  the  warfighter  while 
meeting  a  target  budget. 

Sensitivity  Analysis 

The  weights  for  each  evaluation  measure  and  value  were  varied  while  holding  the 
other  measures  at  a  constant  ratio.  From  Kirkwood’s  discussion  reviewed  in  Chapter  2, 
the  following  equation  was  developed  for  varying  weights: 
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for  all  w  k  where  k  *  j 


Equation  5 


where  w  is  defined  as  a  local  weight  in  a  tier  of  the  value  model,  k  is  the  index  to  the 
weight  to  be  kept  at  a  constant  ratio,  j  is  the  index  to  the  weight  being  varied,  and  w°  is 
the  baseline  weight  for  the  weight  being  kept  at  constant  ratio.  The  results  were  graphed 
showing  changes  in  which  programs  are  most  valuable  depending  on  weights.  The 
analysis  will  quantify  how  robust  the  decision  analysis  results  are. 

Scoring  Lessons  Learned 

In  most  cases,  the  program  managers  who  were  scoring  were  able  to  estimate 
overall  ratios,  but  found  it  difficult  to  estimate  a  value  for  a  system  with  the  technology 
and  a  value  for  one  without  the  technology.  The  best  method  to  overcome  this  would  be 
to  develop  or  make  use  of  existing  simulation  software  to  better  estimate  these 
parameters.  Scoring  the  evaluation  measures  under  Reach  was  difficult  and  the 
evaluation  measures  under  Power  were  even  more  difficult. 

In  fact,  under  Power,  the  Program  Managers  reported  they  were  guessing  at  the 
values  for  the  measures.  They  knew  that  one  program  should  score  higher  than  another, 
but  they  did  not  feel  confident  in  quantifying  an  actual  scoring  level.  It  might  help  to 
change  the  name  of  the  “Effectiveness”  to  “Target  Survival”  since  the  evaluation  measure 
is  really  using  the  probability  of  the  target  surviving.  Additionally,  it  might  help  to 
change  the  name  of  “Survival”  to  “Platform  Pk”  since  this  evaluation  measure  considers 
the  probability  of  killing  a  friendly  unit,  rather  than  the  actual  survival  of  the  unit.  The 
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concepts  are  the  complements  of  what  is  specified  by  the  evaluation  measure  name,  but 
considerable  confusion  resulted  in  attempting  to  explain  these  concepts. 

Scoring  using  the  Air  Vehicles’  value  model  required  10  hours  for  the  seven 
programs  scored.  The  scoring  was  completed  with  four  program  managers,  since  some 
manage  more  than  one  program.  Scoring  in  small  groups  is  important  because  it  allows 
the  program  managers  to  communicate  details  about  their  program.  These  details  are 
useful  for  the  analyst  to  help  develop  insights  in  the  results.  The  program  managers  of 
the  programs  were  the  best  people  to  score  their  programs,  although  some  tool  to  help 
them  estimate  specific  probabilities  would  be  very  helpful  and  might  decrease  the  time 
for  scoring. 

Suitability  of  the  Air  Vehicles’  Model 

Three  questions  can  be  asked  about  the  Air  Vehicles’  model  as  to  its  usefulness. 
First,  does  the  model  represent  all  that  is  important  (collectively  exhaustive)  about 
aerospace  R&D?  Second,  is  this  model  comparable  to  what  has  been  done  before? 

Third,  are  there  any  problems  with  the  model  limiting  its  capability  to  evaluate  Air 
Vehicles’  R&D?  Clearly,  other  specific  questions  can  be  asked  such  as:  1)  is  the  model 
doing  what  it  was  intended  to  do  (verification),  2)  is  the  model  accurate  for  its  intended 
use  (validation),  and  3)  should  the  model  be  used  by  Air  Vehicles  (accreditation)  (DoD 
Directive  5009.59,  1994).  However,  verification,  validation  and  accreditation  (VV&A) 
requires  independent  third  party  involvement,  at  least  to  some  degree,  to  eliminate  bias 
(Defense  Modeling  and  Simulation  Office,  1996:  2-1 1-  2-12).  Hence,  only  the  first  three 
questions  will  be  considered. 
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Table  8  summarizes  how  the  Air  Vehicles’  value  model  covers  dimensions  of 


R&D  identified  in  Table  1  of  Chapter  2  and  provides  the  information  needed  to  answer 
the  first  question.  Nearly  every  dimension  was  covered  either  directly  or  indirectly  by 
Air  Vehicles’  values  with  the  exception  of  training  and  educating  people.  Training  and 
education  of  people,  one  result  of  basic  research  (Martin,  1996:  346)  was  not  part  of  the 
value  model  at  Air  Vehicles’  request.  It  belongs  to  a  separate  analysis  effort,  which 
would  concentrate  on  what  is  important  about  developing  Air  Force  R&D  people. 
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R&D  Dimension 

Measurment 

Aerospace  Specific  Measurements 

All  measures  of  “Gain,”  “Sustain”  and  “Lethality.” 

Contribute  to  Society 

“Support  the  Warfighter”  Defense  of  society  is  a 
contribution  to  it. 

Evolutionary  versus  Revolutionary 

“Expand  our  Frontier”  and  maximizing  applicable 
“Gain  ”  “Sustain”  and  “Lethality”  metrics. 

Improve  Capability 
-Theoretical 

-  Empirical 

-  Methodological 

Indirectly  measured  by  how  it  contributes  to  a 
weapons  system  in  “Gain,”  “Sustain”  and 
“Lethality.” 

Performance  versus  Technical 

Performance  parameters  are  measured  while 
technical  parameters  are  not  measured.  Technical 
parameters  are  the  means  of  accomplishing  the 
performance  parameters. 

Programs 

The  Air  Vehicles’  value  model  scores  programs. 

Qualitative  and  Quantitative 

-  Quantitative-objective 

-  Quantitative-subjective 

-  Qualitative-subjective 

Values,  discrete  evaluation  measures,  and 
continuous  evaluation  measures  were  used  in  the 
value  model  and  can  be  categorized  by  these 
dimensions. 

Risk 

Programmatic  Risk  measured,  but  this  is  a  limited 
way  of  accomplishing  the  task.  Value  focused 
thinking  uncertainty  methods  would  improve  this. 

Scientific  Knowledge 

Measured  indirectly  by  how  it  improves 
performance  or  decreases  cost. 

Technological  Change 

All  measures  of  “Gain,”  “Sustain”  and  “Lethality” 
compare  a  reference  system  with  the  proposed  new 
system.  Several  measures  under  Technological 
Change  show  this,  also.  This  parameter  is 
measured  indirectly  as  “change  in  warfighter 
technological  support.” 

Technological  Trajectories 

-  Cost  Reduction 

-  Incidence  of  Capitol  Cost 

-  Space  Time  Dimension  of 
goods  and  services 

-  Incidence  of  Specific 

Resource  Consumption 

-  Lifetime 

-  Size/scale  effects 

Measured  as  follows: 

-  Cost  Reduction:  “Afford” 

-  Incidence  of  Capitol  Cost:  Indirectly  -  “Afford” 

-  Space  Time  Dimension:  Indirectly  through 
“Gain,”  “Sustain”  and  “Lethality” 

-  Resource  Consumption:  Payload/Range, 
Regeneration  and  Availability 

-  Lifetime:  “Achieve/Extend  Life” 

-  Size/scale  effects:  Indirectly  -  “Afford” 

Train  and  Educate 

Not  measured  -  out  of  scope  of  model. 

Military  Specific 

-  U.S.  Air  Force  Strategic 

Planning 

-  U.S.  Air  Force  Doctrine 

-  Historical  Analysis  of  War 

Measured  by  “Be  Efficient,”  “Simplify 

Operations,”  and  “Win” 

Table  8:  Dimensions  covered  by  Air  Vehicles  Value  Model 
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The  other  area  that  is  covered  in  a  limited  way  is  risk.  This  area  could  be 
improved  in  the  future  using  probabilistic  methods  available  to  Value-Focused  Thinking 
(Kirkwood,  1997:  Chapter  7).  Probabilistic  analysis  was  not  implemented  in  the  Air 
Vehicles’  model  due  to  lack  of  needed  resources. 

The  conclusion  of  this  survey  using  the  literature  is  that  the  Air  Vehicles’  value 
model  is  collectively  exhaustive  in  that  it  covers  what  is  important  to  the  Air  Force 
warfighter.  The  question  of  whether  or  not  the  value  model  was  collectively  exhaustive 
was  also  asked  of  the  Integrated  Concept  Leaders,  the  Strategic  Integrated  Process  Team 
members,  and  the  senior  management  of  Air  Vehicles.  They  have  all  agreed  that  the 
model  is  collectively  exhaustive,  to  the  best  of  their  knowledge.  In  the  future,  it  is 
recommended  the  model  be  shown  to  relevant  Major  Commands  (MAJCOMs)  in  the 
USAF  to  determine  if  they  agree. 

The  second  question  of  whether  the  Air  Vehicles’  model  is  as  good  as  previous 
models  can  be  addressed  using  the  data  in  Table  9.  It  identifies  characteristics  of  the  Air 
Vehicles  value  model  and  optimization  capabilities  as  documented  by  Cetron,  Martino, 
and  Ropcke;  Bretschneider  and  Sounder.  The  Air  Vehicles  model  again  covers  almost  all 
areas  identified  or  is  flexible  enough  that  the  missing  capabilities  could  be  added  to  the 
model.  The  model’s  weaknesses  are  directly  related  to  its  complexity.  Because  R&D  is 
multidimensional,  it  would  be  difficult  to  simplify  the  model  while  maintaining  accuracy 
in  the  results. 
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R&D  Model  Characteristic 

Explanation 

-  Utility  Measure 

-  Orthogonality  of  Criteria 

-  Sensitivity 

-  Rejected  Alternatives  Retention 

-  Classification  Structure 

-  Graphical  Display 

-  Computerized 

-  Computer  Program 

-  Multiple  objective 

-  Hierarchical  Structure 

-  Basic/ Applied  Research 

-  Priority  Decisions 

-  Low  set  up  and  computer  Costs 

-  Budget  Limits 

-  Termination  Decisions 

-  Competitor  Effects 

-  Optimization  Analysis 

-  Strategy  Needs 

-  Quick  computer  run  time 

The  Air  Vehicles  value  model  with  optimization 
has  these  characteristics.  Note:  Competitor 
effects  for  the  warfighter  can  be  defined  as  an 
adversary’s  predicted  reaction  to  new 
technology.  Optimization  analysis  is  simplified 
in  this  model. 

Probability  of  Success,  Market  Risk, 
Technical  Risk,  Premises  uncertainty 

Market  risk  and  premises  risk  is  not  applicable. 
Probability  of  success  and  technical  risk  are 
capture  by  “Programmatic  Risk”  measure. 
Technical  risk  is  assumed  constant. 

-  Manpower  and  Facility  Limits 

-  Multiple  Time  Periods 

-  Project  Interdependencies 

-  Strategies  or  future  scenario 
analysis 

-  Improved  User  friendliness 

The  Air  Vehicles  value  model  with  optimization 
could  have  these  options  added  on. 

-  Some  variables  not  familiar 

-  Requires  a  computer 

-  Requires  an  analyst 

-  Some  special  interpretation  needed 

-  Not  capable  of  simulation  or 
scheduling  analysis 

-  Requires  large  amounts  of  data  and 
extensive  time  of  personnel. 

The  Air  Vehicles  value  model  with  optimization 
has  these  weaknesses. 

Table  9:  Characteristics  of  the  Air  Vehicles  Model  compared  to  R&D  Model 

Features  in  the  Literature 


The  third  question  of  whether  or  not  the  model  is  too  limited  to  properly  describe 
R&D  issues  can  be  determined  using  Baker  and  Freeland’s  criteria  (Baker  and  Freeland, 
1975:  1165).  They  identified  seven  limitations  of  benefit  measurement  models  and 
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resource  allocation  models.  The  first  consideration  deals  with  risk  and  uncertainty. 

Value  focused  thinking  with  optimization  allows  in-depth  treatment  of  the  concepts,  but 
the  effort  was  beyond  the  scope  of  this  effort  (see  Kirkwood,  1997:  Chapter  7  for 
example).  Baker  and  Freeland’s  concern  on  treatment  of  multiple  and  interrelated  criteria 
is  difficult  to  handle  with  value  focused  thinking,  since  mutually  exclusive  values  and 
evaluation  measures  are  needed  (Keeney,  1992:  78).  However,  the  builders  of  the  Air 
Vehicles’  model  felt  the  values  and  evaluation  measures  were  mutually  exclusive  and 
covered  what  is  important  to  Air  Vehicles.  The  builders  were  the  analyst  team  for  this 
project,  the  Air  Vehicles’  Integrated  Concept  Leaders,  and  the  Strategic  Integrated 
Concept  Leaders.  Optimization  models  can  also  be  used  to  address  interrelated  criteria 
(Aaker  and  Tyebjee,  1978;  Schmidt,  1993),  although  these  methods  were  not  needed  in 
this  effort. 

Baker  and  Freeland’s  issue  of  maintaining  balance  between  nonmonetary  aspects 
of  R&D  was  addressed  by  value  focused  thinking  with  optimization  and  is  partially  taken 
care  of  by  the  Air  Vehicles’  value  model.  This  is  possible  simply  in  the  identification  of 
the  values  and  the  evaluation  measures,  which  came  from  Air  Vehicles.  Golabi, 
Kirkwood,  and  Sicherman  went  further  and  used  a  clever  mixed  integer  program 
combined  with  decision-maker  interactions  to  balance  constraints  (Golabi,  Kirkwood, 
and  Sicherman,  1981:  176),  but  that  kind  of  effort  was,  again,  beyond  the  available 
resources. 

The  use  of  value  focused  thinking  with  Air  Vehicles’  should  overcome  Baker  and 
Freeland’s  issue  that  models  do  not  take  advantage  of  R&D  experts’  experience  and 
knowledge  since  it  uses  extensive  expert  inputs.  For  examples  of  how  expert  knowledge 
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is  used,  see  Parnell  et.  al.,  1998;  Golabi,  Kirkwood,  and  Sicherman,  1981.  The  concern 
brought  up  by  Baker  and  Freeland  that  R&D  models  are  complex  will  not  be  dispelled  by 
using  this  type  of  model  as  value  focused  thinking  can  result  in  complex  formulations. 
For  examples  of  how  complex  a  model  can  become,  see  Parnell  et.  al.,  1998;  Golabi, 
Kirkwood,  and  Sicherman,  1981.  Baker  and  Freeland’s  last  issue,  identifying  problems 
of  treatment  of  time  variant  data  and  criteria,  is  handled  in  this  analysis  by  using 
quantitative-subjective  and  qualitative-subjective  measures  (Werner  and  Sounder,  1997: 
34-37)  with  discrete  evaluation  measures  (Clemen,  1996:  80). 

Methodology  and  Modeling  Summary 

Reasons  for  selecting  value  focused  thinking  to  select  technology  for  Air  Vehicles 
were  presented.  Next,  the  Air  Vehicles’  value  model  was  developed  and  presented. 
Finally,  a  subset  of  Air  Vehicles’  programs  was  presented,  program  evaluation  methods 
were  discussed  and  the  suitability  of  the  model  was  explored. 
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Chapter  4:  Results  and  Analysis 


Introduction 

With  the  Air  Vehicles’  value  and  optimization  models  discussed  and  developed, 
analysis  is  performed.  First,  the  Air  Vehicles’  Value  model  is  applied.  This  allows 
ranking  of  programs  and  determination  of  value  gaps  or  identification  of  where  programs 
do  not  score  highly.  Next,  sensitivity  analysis  is  performed  and  several  topics  are 
examined.  First,  how  stable  the  value  model  is  with  changes  in  weights  is  examined. 
Second,  a  possible  way  of  quantifying  how  technologist  and  operators  in  the  USAF  value 
technology  was  proposed.  Next,  the  importance  of  Technological  Superiority  to  Air 
Vehicles  is  discussed.  Additionally,  what  the  different  Integrated  Concepts  value  about 
making  gains  in  new  technology  is  looked  at.  Finally,  some  critical  uncertain  variables 
requiring  further  study  are  identified. 

In  the  final  section  of  the  Chapter,  optimization  techniques  are  applied  to 
maximize  Air  Vehicles’  support  for  the  warfighter  for  various  target  budgets.  In 
preparation  for  optimization,  the  change  in  warfighter  support  due  to  technology  is 
quantified  and  examples  of  meaningful  statements  using  the  results  are  provided.  Using 
budget  constraints,  programs  are  optimized  with  an  “all  or  nothing”  funding  assumption 
or  with  an  “allowance  of  partial  funding”  assumption. 

Value  of  Air  Vehicles  Technology 
Ranking  of  a  subset  of  Air  Vehicles’  Programs 

The  Air  Vehicles’  value  model  was  used  to  analyze  the  benefit  to  the  warfighter 
of  seven  technology  programs  currently  in  progress  or  planned  in  the  near  future.  Table 
10  shows  the  overall  value  scores  calculated  for  the  programs  in  rank  order  and  it  shows 
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hypothetical  “programs”  which  document  the  highest  possible  score,  the  lowest  possible 
score,  and  the  “No  Change”  alternative.  “Benefit”  is  the  score  each  program  received  for 
supporting  the  warfighter.  The  “Program  Coding”  is  an  arbitrary  label  for  each  program 
that  simplifies  references. 


Air  Vehicles’  Program 

Benefit 

Ranking 

Program 

Coding 

Space-Based  System-of-Systems 
Advanced  Development  Demo  & 
Validation 

48 

1 

F 

Actuation  Health-Monitoring 
Systems  (HMS) 

46 

.2 

E 

MTV 

45 

3 

G 

IT  AC 

WEM 

4 

C 

SITE-M 

42 

5 

D 

Weapons  Bay  Noise  Suppression 

34 

6 

A 

Composite  Repair  of  Aircraft 
Structures 

22 

7 

B 

No  Change 

10 

N/A 

N/A 

Table  10:  Overall  Scores  and  Relative  Rankings  for  Seven  Air  Vehicles’ 

Programs 

The  program  that  provides  the  most  benefit  to  the  warfighter  (using  the  value 
model)  was  determined  to  be  the  “Space-Based  System-of-Systems  Advanced 


Development  Demonstration  and  Validation.”  All  of  the  programs  were  better  than  the 
“No  Change”  alternative,  which  indicates  they  are  all  making  improvements  over  the 
current  deployed  state  of  the  art.  Hence,  this  result  shows  that  the  analyzed  programs 
have  the  potential  of  making  a  substantial  contribution  to  the  warfighter. 

Identifying  Technological  Gaps  using  the  Value  Model 


The  results  from  the  value  model  can  be  used  to  determine  if  there  are  values  not 


covered  by  the  programs.  With  an  exhaustive  list  of  programs,  if  a  value  consistently 
scores  poorly,  identification  of  the  poorly  scoring  values  is  the  first  step  to  fixing  the 
problem.  If  a  value  is  missed  or  consistently  scores  poorly,  programs  can  be  modified  to 
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perform  better  or  new  programs  can  be  formulated  to  meet  the  shortfall.  Chapter  2  listed 
some  methods  of  creatively  generating  new  programs. 

Due  to  the  small  sample  size  of  Air  Vehicles  programs  scored,  an  analysis  of 
value  gaps  can  only  be  illustrated.  It  will  be  demonstrated  by  reviewing  the  programs  in 
a  few  areas  and  observing  shortfalls.  A  complete  set  of  graphical  results  for  the  analyzed 
Air  Vehicles’  programs  can  be  found  in  Appendix  I:  Graphical  Sub-value  Contribution  to 
each  Value  and  can  be  found  in  the  Excel  file  “Ranking  Graphs.xls.”  A  review  of 
hundreds  of  programs  requires  some  development  of  automated  tools  to  speed  up  the 
process. 

The  structure  of  the  value  model  makes  it  is  possible  to  determine  the  amount  of 
value  contributed  by  sub-values  that  are  linked  to  a  higher  tier  value.  Figure  1 1  shows 
the  contribution  made  by  each  of  the  seven  programs  scored  for  Air  Vehicles  along  with 
the  “No  Change”  hypothetical  program  for  comparison.  Any  program  that  does  not 
contribute  at  least  as  much  value  as  the  “No  Change”  program  is  not  improving  the 
deployed  state  of  the  art  for  Air  Vehicles,  although  it  might  be  funded  for  safety  or  cost 
reasons.  Clearly,  all  of  the  programs  in  the  notional  set  are  advancing  technology  for  the 
warfighter  at  least  some  amount. 

To  begin  the  analysis,  the  programs  can  be  visually  compared  with  each  other  and 
with  the  “Hypothetical  Best  Case”  program  to  see  what  area  is  scoring  lower  than 
desired.  For  example,  all  of  the  programs  score  under  “Reach,”  but  none  of  the  programs 
come  close  to  the  “Hypothetical  Best  Case.”  To  find  out  why  this  is  true,  the  next  value 
in  the  value  model  should  be  examined. 
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In  this  case,  Reach  is  divided  into  “Gain”  and  “Sustain.”  Figure  12  shows  the 
contributions  each  program  makes  to  “Gain”  and  “Sustain”  under  Reach.  Clearly,  some 
of  the  “Gain”  programs  score  quite  high  (for  example  consider  Space  Based  System-of- 
Systems  Advanced  Dv  Demo  &  Validation).  However,  under  sustain,  all  of  the  programs 
scored  much  lower  than  the  possible  “Hypothetical  Best  Case.” 

Figure  13  shows  each  program’s  contribution  made  to  “Sustain.”  Out  of  the 
seven  programs  analyzed,  three  show  contributions  to  “Sustain.”  Visual  inspection  of 
Figure  13  shows  that  no  programs  scored  in  “Missions  Enabled,”  “Safety,”  or  “Weight 
Reduction.”  Additionally,  the  programs  did  not  score  extremely  high  in  “Achieve/Extend 
Life.”  If  it  were  found  that  none  of  the  over  100  Air  Vehicles’  programs  scored  in  these 
areas,  it  would  become  important  to  create  or  modify  programs  to  score  in  the  areas. 
Otherwise,  not  all  of  Air  Vehicles’  values  would  be  achieved.  If  the  decision-makers  felt 
after  analysis  that  these  values  were  not  important  after  all,  then  they  should  be  dropped 
from  the  value  model  and  the  section  of  the  model  should  be  reworked  as  needed. 
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Figure  11:  Value  Contributions  for  “Support  the  Warfighter’ 


Figure  12:  Value  Contributions  to  “Reach” 
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Sensitivity  of  the  Air  Vehicles’  Value  Model 

Sensitivity  analysis  was  accomplished  on  all  of  the  weights  for  the  values  and 
evaluation  measures.  The  graphical  results  can  be  found  in  Appendix  J:  Sensitivity 
Analysis  of  all  Values  and  Evaluation  Measures  and  can  be  found  electronically  in  the 
Excel  file  called  “Sensitivity  Analysis.xls.”  In  the  discussion  that  follows,  all  graphs  use 
the  key  shown  in  Figure  14.  Variation  in  weights  was  calculated  for  each  program  score 
for  Air  Vehicles  and  graphed.  “Set  Weight”  represents  the  baseline  weight  selected  by 
Air  Vehicles  for  the  value  or  evaluation  measure  under  examination.  This  value  allows 
comparison  of  changes  to  weights  with  the  baseline.  The  “No  Change”  alternative  is 
graphed  as  a  reference.  Any  program  scoring  lower  than  the  “No  Change”  line  is  not 
improving  the  deployed  state  of  the  art  in  technology  for  the  warfighter. 
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Figure  14:  Sensitivity  Analysis  Key 


Analysis  of  “Support  the  Warfighter” 

Figure  15  shows  the  results  of  varying  the  weight  for  “Reach”  from  its  lowest 
possible  value  to  its  highest  possible  value.  If  the  weight  is  lowered  from  its  baseline 
value,  there  is  no  change  in  the  ranking  of  the  alternatives  until  an  approximate  weight  of 
0.15  is  reached.  When  this  happens,  the  MTV  program  changes  position  with  the 
Actuation  Health  Monitoring  Program.  Other  then  this,  no  other  changes  take  place. 

If  the  weight  is  increased  from  its  baseline  value,  the  “Weapons  Bay  Noise 
Suppression”  program  becomes  more  important,  and  at  a  weight  of  0.6,  it  scores  better 
than  “SITE-M,”  “MTV”,  and  “IT AC.”  As  the  current  weight  for  “Reach”  is  0.28,  the 
importance  of  “Reach”  would  have  to  double  or  be  reduced  by  nearly  half  before  any 
ranking  changes  take  place.  The  conclusion  is  a  major  policy  shift  would  be  needed  in 
Air  Vehicles  before  the  programs  would  change  their  ranks. 
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Table  1 1  shows  the  thresholds  where  changing  the  weight  of  the  top  tier  values 
results  in  a  change  in  the  ranks  of  the  scored  programs.  The  discussion  that  follows 
examines  where  changes  in  the  rankings  of  the  programs  change  to  determine  how  the 
ranks  change  based  on  the  weights  for  the  top  tiered  evaluation  measures.  The  purpose  of 
this  is  to  determine  how  sensitive  Air  Vehicles’  value  model  is  to  the  weight.  If  large 
changes  take  place  for  small  changes  in  the  weights,  the  model  would  be  extremely 
sensitive  and  Air  Vehicles  would  have  to  be  extremely  accurate  in  determining  the 
weights.  If  the  opposite  is  true,  then  there  would  be  room  for  errors  in  subjective 
judgement,  which  would  increase  the  confidence  in  the  modeling  results. 
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Value 

Lower 

Threshold 

Percent 

Decrease 

Upper 

Threshold 

Percent 

Increase 

Reach 

0.28 

0.15 

46% 

0.65 

57% 

Awareness 

0.04 

0 

100% 

0.1 

60% 

Power 

0.09 

0.02 

78% 

0.15 

40% 

Technological 

Superiority 

0.58 

0.43 

26% 

0.73 

21% 

Table  11:  Approximate  Weight  changes  that  first  re-rank  Programs  (Note  - 
Weights  do  not  add  up  to  one  due  to  round  off) 


“Awareness”  has  no  lower  threshold  where  the  programs  change  their  ranks,  but 


would  have  to  be  increased  60%  before  programs  would  change  their  rankings.  Only 
“Actuation  Health  Monitoring  System”  scored  under  “awareness.”  The  threshold 


represents  the  point  where  the  program  becomes  the  most  valuable  program  in  the 
programs  scored  for  Air  Vehicles.  Because  Air  Vehicles  is  not  the  primary  Directorate 
for  developing  “Awareness”  in  AFRL,  it  is  highly  unlikely  this  value  would  increase 
drastically  in  weight. 


“Power”  has  a  lower  threshold  of  0.02  for  weight  where  “IT AC”  and  “SITE-M” 
ranked  fourth  and  fifth  respectively,  change  places.  Starting  at  the  upper  threshold  of 
0.15,  “MTV,”  “Actuation  Health  Monitoring  Systems,”  and  finally  “Space  Based  System 
of  Systems  Advanced  Development  Demonstration  and  Validation”  programs  become 
overwhelmed  by  the  importance  being  placed  on  “Power.”  Because  Air  Vehicles  is  not 
the  primary  Directorate  in  AFRL  for  developing  aerospace  power,  it  is  unlikely  this  value 
would  ever  be  weighted  so  heavily. 

“Technological  Superiority”  has  a  lower  threshold  of  0.43,  where  “MTV”  and 
“IT AC”  ranked  third  and  forth,  respectively  change  places.  A  the  upper  threshold  of  0.73 
for  weight,  “Actuation  Health  Monitoring”  and  “MTV”  ranked  second  and  third  change 
places.  At  nearly  the  same  weight,  “IT AC”  and  “SITE-M,”  ranked  fourth  and  fifth 
respectively,  change  places. 
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Overall,  if  the  programs  do  not  change  ranks  over  a  wide  range  of  changes  in  the 
weights  for  the  Air  Vehicles’  value  model.  Air  Vehicles  does  not  have  to  perfectly 
estimate  the  weights  for  the  value  model.  Further,  in  the  weight  ranges  documented  in 
Table  1 1 ,  at  no  time  did  the  top  ranked  program  “Space  Based  System  of  Systems 
Advanced  Development  Demonstration  and  Validation”  ever  change  positions.  Hence,  it 
can  be  confidently  stated  that  this  program  provides  the  most  value  to  the  warfighter  of 
any  of  the  programs  scored. 

Analysis  of  Extremes  of  Technological  Superiority 


Technological  Superiority,  besides  being  the  highest  weighted  value,  shows 
tremendous  importance  to  the  Air  Vehicles’  Directorate.  Figure  16  shows  how  varying 
the  weight  of  Technological  Superiority  changes  the  rankings  and  overall  scores  of  the 
programs  scored.  For  low  weights,  the  ability  to  distinguish  between  the  different 
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programs  becomes  more  and  more  difficult.  As  the  weight  increases,  the  programs 
diverge  significantly  and  the  scores  increase  dramatically. 

Analysis  of  Gain  versus  Sustain  under  Reach 

The  senior  leadership  of  Air  Vehicles  believe  that  starting  about  2002,  Air 
Vehicles  will  be  increasing  its  emphasis  on  gaining  new  capabilities  rather  than 
sustaining  old  ones.  There  is  clear  reason  to  believe  the  weights  for  “Gain”  and  “Sustain” 
will  be  changing.  Sensitivity  analysis  of  this  quantity  can  provide  some  insights  on  what 
this  might  mean. 


As  the  weight  of  “Gain”  under  “Reach”  increases,  the  “Space  Based  System-of- 
Systems  Advanced  Demonstration  and  Validation”  program  will  increase  in  performance. 
The  analysis  shows  most  of  the  other  programs  improving  in  importance.  However,  the 
“Composite  Repair  of  Aircraft”  program  and  the  “Actuation  Health  Monitoring  System” 

will  tend  to  decrease.  This  is  because  the  “Composite  Repair  of  Aircraft”  program  does 
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not  score  under  “Gain”  and  the  “Actuation  Health  Monitoring  System”  scores  higher 
under  “Sustain”  than  it  does  under  “Gain.”  Future  programs  should  attempt  to  maximize 
“Gain”  and  any  scoring  in  “Sustain”  will  help,  but  will  not  be  as  important  as  it  is 
currently. 

Space  versus  UAV  and  Sustainment  Efforts  under  Reach/Gain 

In  meetings  with  the  Integrated  Concept  Leaders,  there  was  disagreement  on  what 
the  weights  should  be  for  the  evaluation  measures  under  Reach/Gain.  The  best  program 
in  all  cases  was  “Space  Based  System  of  Systems  Advanced  Development  Demonstration 
and  Validation.”  This  program  received  all  of  the  possible  value  in  all  categories  except 
“Payload-Range”.  For  more  information  on  why,  see  Appendix  H:  Air  Vehicles’ 
Programs  and  Scoring.  The  “Composite  Repair  of  Aircraft”  program  always  decreased  in 
value  with  increasing  weight  on  any  evaluation  measure  under  “Gain”  since  it  did  not 
score  under  “Gain.” 

Table  12  summarizes  the  trends  for  each  evaluation  measure  when  the  weight  for 
that  particular  measure  is  increased.  The  Space  programs  improved  in  “Availability”  and 
“Regeneration.”  The  UAV  programs  scored  improved  in  “Missions  Enabled”  and 
“Response  Time.”  The  Sustainment  programs  did  not  do  well  under  Gain,  although  the 
“Weapons  Bay  Noise  Suppression”  program  did  show  improvements  in  “Missions 
Enabled.” 
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Program 

Availability 

Missions 

Enabled 

Regeneration 

Response 

Time 

Weapons  Bay  Noise 

Suppression 

(Sustainment) 

Little 

Change 

Increase 

Little  Change 

Decrease 

Composite  Repair  of 
Aircraft  Structures 
(Sustainment) 

Decrease 

Decrease 

Decrease 

Decrease 

Decrease 

IT  AC  (UAV) 

Little 

Change 

Increase 

Little  Change 

Decrease 

Decrease 

(Nonzero 

Score) 

SITE-M  (UAV) 

Decrease 

Increase 

Little  Change 

Increase 

Decrease 

Actuation  health- 
Monitoring  Systems 
(Space) 

Increase 

Decrease 

Increase 

Decrease 

Decrease 

Space-Based 

S  y  stem-of-S  ystems 
Advanced 
Development 
Demonstration  and 
Validation  (Space) 

Increase 

Increase 

Increase 

Increase 

Decrease 

MTV  (UAV) 

Increase 

Decrease 

Increase 

Increase 

Decrease 

(Nonzero 

Score) 

Table  12:  Scored  Air  Vehicle  Program  Trends  under  Reach/Gain  Evaluation 

Measures 


Critical  Uncertainties 

During  scoring  of  the  programs,  some  of  the  evaluation  measures  were  difficult  to 
estimate  scores.  The  Air  Vehicles’  experts  admitted  they  were  guessing  the  evaluation 
measures  for  “Survival”  under  Power/Lethality.  However,  they  felt  the  “SITE-M” 
program  bounded  the  scores  for  the  other  programs  and  were  certain  which  programs 
improved  “Survival.”  Another  evaluation  measure  that  drew  much  discussion  was  the 
“Revolutionary”  metric  under  Technological  Superiority/Expand  our  Frontier.  The  group 
suggested  the  definition  was  not  quite  correct.  They  felt  “Revolutionary”  should  apply  to 
changing  the  paradigm  of  the  warfighter,  not  changing  the  paradigm  of  Air  Vehicles’. 
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The  other  place  for  uncertainty  was  “Programmatic  Risk.”  The  experts  felt  a 
medium  level  of  “Programmatic  Risk”  should  be  closer  to  a  75-25  split  rather  than  a  50- 
50  split.  This  difference  in  opinion  seems  to  be  more  related  to  risk  aversion  (Kirkwood, 
1997:  136).  Some  people  are  risk  adverse  in  that  they  are  willing  to  do  most  anything  to 
avoid  it.  Others  are  risk  seeking  in  that  they  are  willing  to  gamble  with  the  odds  against 
them  when  the  possible  rewards  are  high.  This  difference  in  opinion  on  how  to  define 
“Programmatic  Risk”  may  be  a  manifestation  of  different  risk  attitudes  in  Air  Vehicles 
and  should  be  addressed  directly. 

With  these  concerns  in  mind,  the  scores  in  the  affected  programs  for  each  area 
were  changed  to  see  if  the  overall  ranks  would  change  for  the  programs.  The  “Survival” 
scores  for  “Weapons  Bay  Noise  Suppression,”,  “ITAC,”  and  “MTV”  were  set  to  the  same 
value  as  the  “SITE-M”  program.  The  ranks  did  not  change,  although  the  scores  did 
change  slightly.  Similarly,  the  “Programmatic  Risk”  scores  were  changed.  However,  the 
global  weight  for  this  evaluation  measure  is  a  magnitude  and  a  half  less  than  all  of  the 
other  evaluation  measures,  which  means  this  evaluation  measure  currently  has  no  effect. 

Varying  “Revolutionary”  did  cause  a  major  change  in  the  ranks  of  the  programs. 
Table  13  shows  the  results  of  making  the  changes.  The  conclusion  from  this  analysis  is 
“Revolutionary”  is  a  prime  candidate  for  further  study  and  might  actually  be  better 
analyzed  using  uncertainty. 
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Program 

New 

Ranking 

Weapons  Bay  Noise 
Suppression  * 

Hi 

6 

Composite  Repair  of  Aircraft 
Structures  * 

7 

7 

ITAC* 

1 

4 

SITE-M  * 

2 

5 

Actuation  health-Monitoring 
Systems 

4 

2 

Space-Based  System-of- 
Systems  Advanced 
Development  Demonstration 
and  Validation 

3 

1 

MTV 

5 

3 

Table  13:  Change  in  Ranks  by  Changing  “Revolu 

Marked  with 


ionary”  for  Programs 


The  Cost  of  Creating  Change  in  Warfighter  Technological  Support 

Measuring  the  Change  in  Warfighter  Support  for  a  Program 


It  would  be  incorrect  to  conclude  that  Air  Vehicles  should  only  fund  the  programs 
resulting  in  the  greatest  benefit  to  the  warfighter,  since  the  benefit  achieved  comes  at  a 
price.  “Change  in  warfighter  technological  support”  is  defined  as  the  difference  between 
a  program  score  and  the  “No  Change”  alternative  score.  A  program  that  provides  large 
benefits  for  small  investment  costs  is  preferred  to  one  that  provides  large  benefits  for 
large  investment  costs. 


One  approach  for  determining  the  best  benefit  to  the  warfighter  for  a  particular 
investment  cost  is  to  calculate  the  benefit  to  cost  ratio  (Kirkwood,  1997:  200-206;  Bell 


and  Read,  1974:  36).  Table  14  shows  the  programs  re-ranked  from  the  highest  to  the 
lowest  benefit  per  cost  ratio.  The  best  programs  are  those  that  provide  large  jumps  in 
benefit  at  the  smallest  possible  cost.  In  the  Air  Vehicles’  subset  of  programs,  it  is  clear 
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that  all  of  the  programs  provide  varying  amounts  of  benefit,  but  some  require  larger 
investments  for  the  benefit  attained  than  others. 


Alternative 

Cost 

(Millions 
of  Dollars) 

Warfighter 

Benefit 

Benefit/ 

Cost 

Program 

Code 

Letter 

Change  in 
warfighter 
Technological 
Support 

SITE-M 

2 

41.73 

20.86 

D 

31.78 

Actuation  Health-Monitoring 
Systems  (HMS) 

4 

46.00 

11.50 

E 

36.05 

Weapons  Bay  Noise 

Suppression 

7 

34.00 

4.86 

A 

24.05 

IT  AC 

11 

42.62 

3.87 

C 

32.67 

Composite  Repair  of  Aircraft 
Structures 

6 

22.36 

3.73 

B 

12.41 

Space-Based  System-of-systems 
Advanced  Development  Demo 
&  Validation 

25 

48.03 

1.92 

F 

38.07 

MTV 

25 

44.58 

1.78 

G 

34.63 

Table  14:  Air  Vehicles’  Programs  Ranked  from  highest  to  lowest  Benefit  to 

Cost  Ratio 


However,  the  “Strategic  Position”  in  the  Air  Vehicles’  strategic  states  Air 
Vehicles  will  “deliver  the  best  air  vehicle  technologies  for  aerospace  dominance  against 
all  threats  . . Using  the  benefit  to  cost  ratio  to  rank  the  programs  is  flawed  because  the 
decision-maker  cannot  be  certain  if  the  proposed  programs  are  actually  improving  the 
deployed  state  of  the  art  or  not.  Additionally,  the  benefit  to  cost  ratio  method  using 
“Warfighter  Benefit”  in  Table  14,  measures  the  best  programs  for  aerospace  rather  than 
the  best  technologies  for  aerospace.  While  technological  change  cannot  be  directly 
measured  from  the  Air  Vehicles’  value  model,  what  can  be  measured  is  the  change  in 
warfighting  technological  support,  which  was  discussed  in  Chapter  3.  This  quantity 
compared  with  cost  can  be  found  in  Figure  18. 
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Figure  18:  The  Change  in  Warfighter  Benefit  versus  the  Cost 


A  new  parameter,  called  change  in  warfighter  technological  intensity  is  defined  as 
the  ratio  of  change  in  warfighting  technological  support  to  investment  cost.  Some 
observations  about  this  variable  are  in  Appendix  K:  Intensity  of  Change  in  Warfighter 
Support  due  to  Technology  .  Table  15  shows  the  Air  Vehicles’  programs  scored  in 
descending  order  of  change  in  warfighter  technological  intensity. 

In  Chapter  3,  the  change  in  warfighting  technological  support  was  found  to  be 
measured  on  a  ratio  scale.  Consequently,  using  the  values  in  Table  15,  it  is  possible  to 
make  statements  like  “the  ‘Weapons  Bay  Noise  Suppression’  program  (score  of  24.05) 
provides  nearly  twice  the  change  in  warfighting  technological  support  that  the 
‘Composite  Repair  of  Aircraft  Structures’  program  (score  of  12.41)  provides.”  Similarly, 
it  can  be  stated  that  “the  ‘Space-Based  System-of-Systems  Advanced  Development 
Demo  &  Validation’  (score  of  38.07)  is  expected  to  provide  over  three  times  the 
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technological  advancement  in  Air  Force  warfighting  that  the  ‘Composite  Repair  of 
Aircraft  Structures’  program  (score  of  12.41)  provides.” 


Alternative 

Cost  (Cj) 
(Millions 
of  Dollars) 

Change  in 
Warfighter 
Technological 
Support  (Tj) 

Change  in 
Warfighter 
Technological 
Intensity 

Program 

Code 

Letter 

SITE-M 

2 

31.78 

15.89 

D 

Actuation  Health-Monitoring 
Systems  (HMS) 

4 

36.05 

9.01 

E 

Weapons  Bay  Noise 

Suppression 

7 

24.05 

3.44 

A 

ITAC 

11 

32.67 

2.97 

C 

Composite  Repair  of  Aircraft 
Structures 

6 

12.41 

2.07 

B 

Space-Based  System-of-Systems 
Advanced  Development  Demo 
&  Validation 

25 

38.07 

1.52 

F 

MTV 

25 

34.63 

1.39 

G 

Table  15:  Change  in  Warfighter  Technological  Support  of  Air  Vehicles’ 
Programs  Ranked  from  highest  to  lowest  Technological  Intensity 


The  total  change  in  warfighter  technological  support  can  be  compared  with  the 
total  investment  cost  using  the  change  in  warfighter  technological  intensity  for  ranking 
purposes.  The  result  is  shown  in  Figure  19.  The  decision-maker  can  use  this  method  to 
find  a  good  solution  of  what  programs  should  be  funded.  The  decision-maker  can  choose 
the  funding  level  for  the  portfolio  and  then  can  determine  impacts  the  technology  will 
have  on  improving  the  deployed  state  of  the  art.  As  the  funding  increases,  the  program  at 
the  chosen  funding  level  and  the  programs  at  lower  funding  levels  become  part  of  the 
overall  portfolio  of  programs. 

In  using  the  method,  however,  the  decision-maker  should  realize  that  the  result 
has  many  limitations.  First,  the  portfolio  does  not  have  to  be  optimal  (Kirkwood,  1996: 
200-205;  Bell  and  Read,  1974:  36).  Second,  there  is  no  insight  provided  on  possible 
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portfolio  combinations  between  the  plotted  points.  Third,  the  method  can  only  use  a 
single  constraint  (cost  in  this  case)  and  assumes  that  the  programs  do  not  interact 
(Kirkwood,  1997:  206).  To  be  guaranteed  of  an  optimal  portfolio  of  programs,  linear  or 
integer  programming  techniques  are  required.  This  topic  will  be  examined  next. 


Figure  19:  Total  Change  in  Warfighter  Technological  Support  versus 

Cumulative  Cost  Results 

The  Change  in  Warfighter  Support  due  to  Technology  versus  Program  Cost 
Assuming  “All  or  Nothing ”  Funding 

Comparing  technological  change  with  investment  cost  allowed  the  development 
of  a  possibly  sub-optimal  portfolio  of  programs.  The  first  policy  type  examined  for 
optimization  is  the  case  where  a  program  is  either  fully  funded  or  dropped  from  the 
planned  portfolio.  To  determine  the  optimal  solution,  the  following  binary  or  0-1  math 
program  was  developed: 


99 


Let  x .  =  the  fraction  of  program  i  that  is  funded. 

Let  7]  =  the  change  in  warfigher  technological  support  to  be  created  because  of 
implementation  of  program  i. 

Let  C .  =  the  investment  cost  of  implementing  program  i  (in  millions  of  dollars). 

Let  B  =  the  available  funding  for  a  portfolio  of  programs. 

Let  ie  {A,B,C,D,E,F,G}  which  are  the  programs  to  be  optimized. 

Maximize  Technological  Change  = 

i 

Such  That 

£c,*,  <  B 

i 

Xjg{0,1}  for  alii 

The  values  for  Change  in  Warfighter  Support  due  to  Technology  (TO  and  the 
investment  cost  (Q)  are  taken  from  Table  15.  The  available  budget,  B,  was 
parameterized  to  vary  from  no  dollars  to  $80  million  in  increments  of  $1  million.  The 
$80  million  figure  was  chosen  since  this  is  the  minimum  investment  required  funding  for 
all  seven  of  the  Air  Vehicles’  programs  scored  in  the  study. 

Figure  20  shows  the  optimized  change  in  warfighter  technological  support  that 
can  be  created  versus  all  investment  cost  up  to  the  point  where  all  programs  are  funded. 

It  also  shows  the  change  in  warfighter  technological  support  versus  cost  calculated  using 
the  benefit  to  cost  ratio.  In  this  situation,  the  benefit  to  cost  ratio  method  turned  out  to  be 
optimal  at  the  points  calculated.  The  0-1  math  calculations  show  combinations  of 
programs  can  be  specified  that  provide  optimum  portfolio  policies  at  intermediate  points 
not  found  by  the  benefit  to  cost  ratio  method. 

Figure  21  shows  the  optimized  portfolio  of  programs  that  provides  the  most 
change  in  warfighter  technological  support  for  investments  between  $60  million  and  $80 
million.  A  change  of  $1  million  dollars  difference  in  funding  can  make  a  substantial 
difference  in  which  programs  are  funded.  For  example,  a  drop  in  funding  from  $74 


100 


million  to  $73  million  results  in  “Weapons  Bay  Noise  Reduction”  program  not  being 
funded  in  favor  of  “Composite  Repair  of  Aircraft  Structures.”  Another  drop  of  $1 
million  dollars  results  in  “ITAC”  being  dropped  in  favor  of  funding  both  “Weapons  Bay 
Noise  Reduction”  and  “Composite  Repair  of  Aircraft  Structures.”  In  the  next  section,  the 
assumption  of  “all  or  nothing”  funding  is  relaxed,  allowing  consideration  of  partially 
funding  programs. 


Figure  20:  Optimized  Change  in  Warfighter  Technological  Support  versus 
Investment  Cost  using  0-1  Math  Program 
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Optimization  of 
Scored 


Change  in  Warfighter  Technological  Support  versus  Program  Cost  Assumes 
Partial  Funding  of  Programs 

By  relaxing  the  integer  (all  or  nothing)  requirement,  partial  funding  can  be 
considered.  The  linear  programming  implementation  for  calculating  change  in  warfighter 
technological  support,  is  shown  below: 

Let  x(  =  the  fraction  of  program  i  that  is  funded. 

Let  7j  =  the  technological  change  to  be  created  because  of  implementation  of 
program  i. 

Let  C,  =  the  investment  cost  of  implementing  program  i  (in  millions  of  dollars). 

Let  B  =  the  available  funding  to  fund  a  portfolio  of  programs. 

Let  /  g  {A,  B ,  C,  D,  E,  F,  G}  which  are  the  programs  to  be  optimized. 

Let  L.  =  the  lower  fraction  of  funding  allowed  for  program  i. 

Maximize  Technological  Change  =  Tjjc; 

I 

Such  That 

<B 

i 

L,<x j<1  for  all/ 

The  value  Lj  is  the  lower  bound  that  program  i  can  drop  to  and  still  improve  the  deployed 
state  of  the  art  in  technology.  This  formulation  assumes  that  both  investment  cost  and 
change  in  warfighter  technological  support  can  be  reduced  linearly  until  the  lower  bound 
Li  is  reached.  Specifying  Li  results  in  the  hard  constraint  that  program  i  must  be  funded 
at  least  to  the  level  of  Lj. 

Two  cases  of  the  linear  program  were  run.  First,  a  case  where  L  =  0  was  run  to 
determine  the  maximum  possible  change  in  warfighter  technological  support  for  various 
budget  levels.  Then,  the  following  arbitrary  values  for  L  were  set: 

La  =  0.75,  LB  =  0,  Lc  =  0,  LD  =  0.75,  LE  =  0.75,  LF  =  0,  Lg  =  0.95. 
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Figure  22  shows  the  optimal  change  in  warfighter  technological  support  for 
various  budgets  for  both  partial  funding  cases  and  the  “all  or  nothing”  funding  profile 
shown  in  the  last  section.  Table  16  shows  the  percent  funding  for  the  Air  Vehicles’ 
programs  for  various  funding  levels.  The  result  shows  the  “No  Minimum  Required” 
funding  profile  provides  the  most  total  change  in  warfighter  benefit  at  a  particular  budget 
level.  In  general,  this  is  the  best  and  most  desired  funding  profile  that  is  possible. 
However,  it  is  impossible  to  achieve  this  profile  since  R&D  programs  cannot  be  funded 
below  certain  levels  and  still  gain  any  useful  results. 

The  “Minimum  Required  Partial  Funding”  profile  shows  the  penalties  paid  by 
being  forced  to  fund  some  of  the  projects  at  a  relatively  high  level.  The  profile  stops  at 
$34  million  because  below  $34  million  it  is  impossible  to  meet  all  of  the  funding 
requirements  (the  linear  program  is  infeasible  below  $34  million).  The  “All  or  Nothing 
Funding”  profile  shows  the  profile  for  the  programs  if  the  assumption  is  they  must  be 
100%  funded  or  not  funded  at  all.  With  this  in  mind,  it  appears  there  may  be  times  when 
a  partial  funding  policy  with  positive  lower  funding  limits  and  the  “All  or  Nothing” 
funding  policy  each  provide  more  benefit  than  the  other.  This  is  shown  by  the  fact  that 
some  of  the  “All  or  Nothing”  results  are  higher  than  the  equivalent  “Minimum  Partial 
Funding”  results.  The  reverse  also  takes  place  showing  that  partial  funding  is  sometimes 
the  better  policy  for  maximizing  change  to  warfighter  support  due  to  technology. 
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Total  Change  in  Warfighter  Benefit 


200 


150 


100 


50 


0 


—  No  Minimum  Required 
Partial  Funding 

Minimum  Required 
Partial  Funding 
+  All  or  Nothing  Funding 


“Minimum  Required  Partial 
Funding”  stops  due  to 
infeasibility.  At  $34M  and 
below  it  is  impossible  to  fund 
all  of  the  programs. 


0 


20  40  60 

Budget  Levels  (Millions  of  Dollars) 


Figure  22:  Optimized  Change  in  Warfighter  Technological  Support  versus 
Investment  Costs  using  Linear  Programming 
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Table  16:  Percent  Funding  Profile  for  Minimum  Partial  Funding 


D 

E 

100% 

100% 

100% 

100% 

100% 

100% 

100% 

100% 

100% 

100% 

100% 

100% 

100% 

100% 

100% 

100% 

100% 
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100% 

100% 

100% 

100% 

100% 

100% 

100% 

100% 

100% 

100% 

100% 

100% 

100% 

100% 

100% 

100% 

100% 

100% 

100% 

100% 

100% 

100% 

100% 

100% 

100% 

100% 

100% 

100% 

100% 

100% 

100% 

100% 

100% 

100% 

100% 

100% 

100% 

100% 

100% 

100% 

100% 

100% 

100% 

100% 

100% 

100% 

100% 

100% 

100% 

100% 

100% 

100% 

100% 

100% 

100% 

100% 

100% 

100% 

100% 

100% 

100% 

100% 

100% 

100% 

100% 

100% 

100% 

100% 

100% 

100% 

100% 

100% 

c 

B 

F 

G 

0% 

0% 

0% 

95% 

0% 

0% 

0% 

95% 

2% 

0% 

0% 

95% 

11% 

0% 

0% 

95% 

20% 

0% 

0% 

95% 

30% 

0% 

0% 

95% 

39% 

0% 

0% 

95% 

48% 

0% 

0% 

95% 

57% 

0% 

0% 

95% 

66% 

0% 
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95% 
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100% 
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100% 
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100% 

100% 
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95% 

100% 

100% 

13% 

95% 

100% 

100% 

17% 

95% 

100% 

100% 

21% 

95% 

100% 

100% 

25% 

95% 

100% 

100% 

29% 

95% 

100% 

100% 

33% 

95% 

100% 

100% 

37% 

95% 

100% 

100% 

41% 

95% 

100% 

100% 

45% 

95% 

100% 

100% 

49% 

95% 

100% 

100% 

53% 

95% 

100% 

100% 

57% 

95% 

100% 

100% 

61% 

95% 

100% 

100% 

65% 

95% 

100% 

100% 
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Other  Costs  Besides  Investment  Dollars 


There  are  other  types  of  constraints,  that  are  likely  to  be  important  to  Air  Vehicles 
other  than  the  investment  dollar  amounts.  According  to  the  literature,  typical  constraint 
types  are  budget  constraints,  skill  availability  constraints,  facility  availability  constraints, 
raw  material  availability  constraints,  and  program  balance  and/or  risk  balance  constraints 
(Bretschneider,  1993:  130;  Cetron,  Martins,  and  Roepoke,  1967,  5).  The  constraints  are 
examined  for  how  they  might  be  added  to  the  “all  or  nothing”  funding  math  analysis  or 
the  integer  math  program,  although  similar  types  of  constraints  can  be  added  to  the  partial 
funding  math  analysis. 

One  type  of  constraint  is  the  need  to  balance  programs  among  several  groups.  An 
example  is  to  constrain  the  analysis  such  that  a  minimum  number  of  Integrated  Concepts 
or  basic/applied/development  programs  are  funded.  For  example,  in  the  integer 
programming  example,  if  there  is  a  requirement  to  have  at  least  one  of  each  Integrated 
Concept  program  funded,  the  following  constraints  would  be  added: 

+XB>1 

xc+xD+xc>\  Equation  6 

XE  +X  1 

Another  type  of  constraint  could  deal  with  resources.  If  the  variable  C  (originally 
investment  cost)  is  redefined  to  be  the  amount  of  resource  j  required  to  complete  program 
i  or  Cjj,  then  any  number  of  resources  could  be  accounted  for  by  the  model  (with  in  the 
limits  of  computational  power).  In  this  case,  j  would  be  defined  as  a  resource  type,  which 
would  range  from  one  to  the  total  number  of  resources  being  considered.  Resources 
could  consist  of  budget  constraints,  skill  availability  constraints,  facility  availability 
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constraints,  and  raw  material  availability  constraints.  The  variable  B  would  also  be 
changed  to  have  the  subscript  j,  and  the  constraint  would  look  like: 

X ,Cijxi  -  Bj  Equation  7 

i 

The  variable  Bj  would  now  represent  the  maximum  amount  of  resource  type  j  that  is 
available,  overall.  There  would  be  a  total  of  j  equations  formed  by  this  constraint. 

It  is  also  possible  to  add  constraints  so  the  program  has  multiperiods  for  resources 
and  funding.  A  variable  k  could  be  defined  to  be  the  period  (fiscal  year  say)  a  particular 
resource  is  to  be  provided  for.  Then,  the  following  type  of  constraint  could  be  added: 

X Ci,kxi  -  Bk  Equation  8 

i 

Another  example  of  a  possible  constraint  occurs  if  funding  of  one  program 
requires  another  program  or  a  set  of  programs  to  be  funded.  For  example,  if  program  G 
requires  programs  A,  B,  C,  and  D  (this  is  not  true  in  reality),  then  the  following  constraint 
would  be  formed: 

4xg  ~xa  ~xb  ~xc  ~xd  ^  0  Equation  9 

This  constraint  requires  programs  A,  B,  C  and  D  to  be  funded  if  program  G  is  funded.  If 
program  G  is  not  funded,  programs  A,  B,  C  or  D  can  be  funded  in  any  combination,  with 
no  effect  on  program  G. 

The  constraints  identified  can  be  combined  together  in  the  model.  It  is  also 
possible  to  formulate  other  constraints  that  may  be  unique  to  Air  Vehicles.  More 
information  is  available  in  the  literature  (Bell  and  Read,  1974;  Gear,  1974). 

Summary 

The  Air  Vehicles  model  was  applied  to  analyze  seven  of  Air  Vehicles’  programs. 
The  programs  were  ranked  by  the  benefit  they  provide  to  the  warfighter.  Change  in 
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warfighter  technological  support  was  calculated  for  each  program  and  used  to  identify  the 
impact  the  programs  have  to  the  deployed  state  of  the  art,  to  compare  programs  with  each 
other,  and  used  in  mathematical  optimization  programs.  It  was  found  integer  and  linear 
programming  techniques  can  be  used  together  to  determine  when  to  cut  programs  and 
when  to  scale  back  programs  for  various  budgets. 

Analysis  with  the  Air  Vehicles  Value  model  demonstrated  the  ability  to  find 
values  not  covered  in  a  portfolio  of  programs,  although  with  a  small  sampling  of 
programs,  the  results  are  not  conclusive.  Sensitivity  analysis  showed  the  Air  Vehicles’ 
value  model  does  not  have  large  changes  in  the  alternatives  for  small  changes  in  weights. 
Further,  a  method  for  comparing  technologists’  and  operators’  views  of  the  value  of 
technology  was  proposed.  Finally,  some  evaluation  measures  were  identified  to  be 
uncertain  and  should  be  studied  in  the  future. 
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Chapter  5:  Findings  and  Conclusions 

Introduction 

In  Chapter  1,  it  was  stated  that  R&D  organizations  have  to  justify  their 
expenditures  or  risk  being  dismantled.  Dismantling  of  an  R&D  organization  is  a  serious 
business,  which  could  have  ramifications  far  into  the  future,  depending  on  the 
organization’s  responsibilities.  To  prevent  this,  an  organization  has  to  change  and  adapt 
to  a  constantly  changing  world.  Information  is  critical  to  adapting,  but  information  has  to 
be  organized  to  be  useful.  This  thesis  has  provided  one  way  of  organizing  and  then 
understanding  what  the  information  means. 

The  objective  for  this  work  was  to  answer  this  question: 

“What  direction  in  technology  R&D  should  the  Air  Vehicle  Directorate  take  to  be 

most  consistent  with  Air  Force  values?” 

This  single  question,  which  is  easily  identified,  is  not  simple  to  answer.  The  question 
began  a  research  effort  requiring  extensive  literature  search,  development  of  a 
methodology  with  a  model  and  analysis  to  provide  an  answer. 

This  study  began  with  a  literature  search  to  identify  the  dimensions  of  Aerospace 
R&D  in  the  literature.  Next,  technology  selection  models  were  examined.  Of  the  large 
number  of  models,  Value-Focused  Thinking  was  selected,  due  to  its  theoretically  sound 
foundations  and  because  of  its  unique  ability  to  allow  “out  of  the  box”  thinking. 
Optimization  models  were  also  selected  because  they  allow  development  of  “best” 
program  portfolios.  The  literature  search  was  wrapped  up  by  examining  characteristics  of 
past  technology  selection  models  to  understand  their  strengths  and  weaknesses. 

The  study  next  considered  a  methodology.  It  began  by  setting  up  an  overall 
framework  for  the  study  and  resulted  in  a  model.  It  was  learned  that  Air  Vehicles’ 
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primary  mission  is  to  support  the  USAF  warfighter  with  dominant  technology.  Besides 
developing  improvements  that  provide  the  warfighter  an  advantage  on  the  battlefield,  Air 
Vehicles’  also  has  to  be  technologically  superior  to  any  adversary  the  United  States  might 
face  today  or  in  the  future.  These  concepts  are  what  is  important  to  Air  Vehicles.  In  the 
process,  concepts  like  “deployed  state  of  the  art”  were  quantified  and  issues  peculiar  to 
R&D  such  as  trading  off  technological  attributes  were  identified.  Methods  of  measuring 
change  in  warfighter  technological  support  were  developed.  Most  importantly,  the 
development  of  the  Air  Vehicles’  model  took  place  with  the  Air  Vehicles’  managers  and 
technical  experts. 

Analysis  of  program  results  identified  what  programs  provided  the  most  benefit  to 
the  warfighter  and  those  that  provided  less.  A  method  for  identifying  values  not  scoring 
in  the  alternatives  was  demonstrated.  In  the  process  of  understanding  this,  however,  it 
was  realized  that  change  in  warfighter  technological  support  could  now  be  measured  and 
used  to  draw  conclusions.  Optimization  methods  were  used  to  show  the  best  method  of 
minimizing  the  impacts  of  budget  constraints.  Finally,  analysis  of  the  value  model 
showed  the  model  was  stable  over  a  range  of  weights  on  the  values.  A  model  was 
proposed  to  explain  how  operators  in  the  USAF  might  see  the  value  of  technology  and  to 
show  how  Air  Vehicles’  sees  technology. 

Conclusions 

It  is  possible  to  identify  dimensions  of  R&D  and  aerospace  from  the  literature. 
From  these  results,  a  value  model  was  developed  that  Air  Vehicles  was  able  to  provide 
inputs  to  modify  and  improve  the  model.  The  model  was  used  to  identify  the  change  in 
support  to  the  warfighter  due  to  technology  and  optimization  was  applied  to  maximize 
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the  support  to  the  warfighter  for  various  budget  constraints.  The  model  was  found 
relatively  insensitive  to  fluctuations  in  the  weights.  Use  of  the  R&D  literature  showed 
the  model  was  adequate  for  the  job  of  selecting  technology,  although  more  work  could  be 
done  in  the  area  of  uncertainty.  The  model  was  used  successfully  to  analyze  sever 
programs  for  Air  Vehicles. 

Recommendations 

It  is  recommended  Air  Vehicles  should  implement  this  method  for  selecting 
programs.  In  the  process,  Air  Vehicles  should  verify,  validate  and  accredit  the  model. 
With  the  value  model  developed,  Air  Vehicles  can  score  their  programs,  with  their 
technical  experts,  and  determine  the  value  of  their  programs.  From  there,  it  is 
recommended  that  the  integer  programming  techniques  and/or  the  linear  programming 
techniques  be  used  to  identify  the  best  portfolio  of  programs.  Additionally,  Air  Vehicles 
should  implement  other  modeling  techniques  that  would  help  them  to  better  estimate  the 
scores  for  the  value  model. 

Future  Research 

This  effort  should  be  expanded  to  the  other  Directorates  of  AFRL  and  should  be 
developed  for  AFRL  overall  use.  Originally  AFRL  requested  this  effort,  and  this  study 
was  completed  as  a  first  step  toward  helping  AFRL.  One  approach  to  completing  this  for 
AFRL  would  be  to  approach  the  other  nine  Directorates  and  add  their  values  to  the  Air 
Vehicles  hierarchy.  Undoubtedly,  the  hierarchy  will  change,  especially  at  the  lower 
levels.  The  hierarchy  should  be  built  in  a  modular  fashion.  Representatives  from  each 
Directorate  would  be  needed  to  represent  their  Directorate  and  to  identify  resources  from 
their  Directorate  to  develop  the  model.  Next,  evaluation  measures  would  have  to  be 
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determined  for  the  other  Directorates.  Then,  AFRL  would  be  requested  to  weight  the 
upper  levels,  while  the  Directorates  would  have  to  weight  the  lowest  levels  in  the  value 
model.  Finally,  programs  could  be  scored. 

The  dimensions  of  R&D  should  be  explored  more  thoroughly.  While  extensive 
literature  searching  allowed  identification  of  these  dimensions  in  the  first  place,  it  is 
realized  there  are  other  sources  available  that  could  enrich  the  concepts.  Further,  it  would 
be  interesting  to  examine  other  fields  of  R&D  such  as  propulsion. 

To  allow  development  of  more  creative  alternatives,  aeronautical  experts  need  to 
get  together  and  develop  a  structured  approach  to  explore  what  has  not  yet  been  done. 

One  structured  approach  might  be  to  develop  a  means  hierarchy  for  the  Air  Vehicles’ 
value  model.  Another  idea  is  to  develop  a  value  model  to  identify  what  is  important 
about  the  technical  aspects  of  aeronautics.  This  type  of  model  might  answer  the  question 
“what  is  possible  to  develop  in  the  field  of  aeronautics.”  Alternatives  identified  could  be 
used  in  the  Air  Vehicles  value  model  to  determine  the  value  to  the  warfighter.  This 
exploration  would  also  have  to  identify  possible  military  applications  for  the  concepts. 

The  Air  Vehicles’  value  model  should  be  improved  to  cover  uncertainty.  It  was 
determined  that  identifying  what  revolutionary  a  technology  is  can  be  difficult.  Along 
with  revolutionary,  another  variable  that  should  be  revisited  for  uncertainty  is 
programmatic  risk.  Although  this  was  weighted  extremely  low,  the  result  could  change 
in  the  future. 

Methods  of  more  accurately  predicting  change  in  warfighter  technological  support 
should  be  examined.  With  the  variable  defined,  there  may  be  other  theoretical 
considerations  of  how  it  might  be  applied,  measured  or  derived.  The  concept  of 
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technological  intensity  might  also  be  explored.  It  would  be  extremely  helpful  if  a  way 
could  be  found  to  more  quickly  measure  this  quantify  for  technology. 

The  use  of  alternative  future  scenarios  could  easily  be  applied  to  Air  Vehicles.  In 
finding  out  weights  from  the  Air  Vehicles  senior  decision-makers,  it  was  learned  that  the 
importance  of  Sustain  was  expected  to  diminish  around  2002  while  Gain  would  increase 
(under  Reach).  This  is  a  clear  indication  that  alternative  futures  should  be  examined,  or 
at  least  the  time  horizon  being  examined  directly  changes  the  weights  on  the  value  model. 

The  Excel  models  for  implementing  the  Air  Vehicles  model  can  be  somewhat 
awkward.  Specifically,  adding  new  alternatives  can  be  difficult.  Any  person  who 
develops  a  model  in  Excel  for  value  models  and  math  programming  should  first  examine 
this  one  to  see  where  improvements  might  be  made.  Documentation  for  the 
implementation  can  be  found  in  the  Appendices. 

The  assumptions  made  that  programs  do  not  interact  should  be  relaxed.  This 
assumption  is  convenient  for  the  analyst,  but  is  not  true,  in  practice.  In  the  literature, 
models  were  identified  that  could  overcome  this  limitation.  More  research  is  needed. 

The  Air  Vehicles  Value  Model  should  be  used  to  score  a  much  larger  number  of 
programs,  if  not  all  of  them,  in  Air  Vehicles.  The  programs  should  be  analyzed  to 
determine  where  value  gaps  are  or  where  higher  scoring  programs  are  needed.  Analysis 
might  turn  up  missing  values  in  the  value  model,  or  may  help  Air  Vehicles  create 
alternatives  that  better  support  the  warfighter. 

Another  possible  area  for  future  work  would  be  to  consider  the  time  value  of 
technology,  which  would  be  similar  in  concept  to  the  time  value  of  money.  As  the  time 
increases  to  complete  a  project,  the  value  of  the  technology  will  tend  to  decrease.  That  is. 
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other  technology  being  developed  will  tend  to  degrade  the  importance  originally  attached 
to  the  program.  This  is  a  possible  area  for  future  research. 

Summary 

The  model  developed  for  Air  Vehicles  provides  a  method  to  answer  the  question 
of  how  Air  Vehicles  should  select  technologies  to  be  most  consistent  with  Air  Force 
values.  The  model  was  developed  with  the  expert  help  of  Air  Vehicles  and  by  using 
documents  in  the  literature.  The  model  was  demonstrated  by  scoring  seven  of  Air 
Vehicles’  programs.  This  work  sets  the  foundation  for  further  work  with  Air  Force 
Research  Laboratory  and  its  nine  other  Directorates. 
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Appendix  A:  USAF  Strategic  Documents  Literature  Search  Results 

The  purpose  of  this  Literature  Search  was  to  determine  if  correlation  existed 
between  proposed  Air  Vehicle  Values  of  "Awareness,  Reach  and  Power"  and  with 
various  Air  Force  Strategic  Planning  Documents.  Correlation  was  determined 
qualitatively  based  on  the  definitions  of  various  Air  Force  Strategic  Planning  Concepts 
and  the  proposed  definitions  of  "Awareness,  Reach  and  Power."  The  conclusions  from 
correlating  the  documents  can  be  found  in  Table  17,  while  a  summary  of  the  correlations 
can  be  found  in  Table  18.  Documents  examined  include  Air  Force  Basic  Doctrine,  Joint 
Vision  2010,  Air  Force  2025,  Global  Engagement,  Air  Force  Strategic  Plan,  New  World 
Vistas,  and  the  Air  Vehicles  Strategic  Plan  (Draft  Working  Copy  #3).  Comparisons  of 
Principles  of  War  defined  in  Air  Force  Basic  Doctrine  was  done  for  the  purpose  of 
showing  coverage  by  the  four  values.  Reasons  behind  correlations  are  subject  to 
individual  interpretation  and  experience. 

A  second  purpose  was  to  identify  a  fourth  value  of  the  same  tier  as  the  other 
proposed  values  because  it  was  believed  the  three  values  by  themselves  were  not 
exhaustive  in  detailing  what  was  important  in  technology  for  achieving  air  and  space 
dominance  or  supporting  the  USAF  warfighter.  This  was  done  by  elimination.  When  the 
Strategic  Planning  Documents  examined  had  concepts  not  covered  by  "Awareness,  Reach 
and  Power,"  these  concepts  were  placed  in  the  "Technology  Leadership"  column.  After 
examining  the  documents,  the  concepts  were  combined  into  like  groups.  Three  groups 
were  found:  a  group  with  technology  values,  a  group  dealing  with  Air  Force  people,  and 
a  group  of  constraints  that  limit  the  technology  programs  chosen  for  development.  The 
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groups  were  named  "Technological  Superiority",  "Air  Force  People"  and  "Constraints", 
respectively. 

The  "Technological  Superiority"  concept  was  identified  as  the  fourth  value.  "Air 
Force  People"  was  determined  to  bring  up  questions  beyond  the  scope  of  this  study.  The 
"Constraints"  are  noted  for  future  reference  in  the  AFIT  Thesis  Effort. 
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Technology 

Leadership 

Definitions 

Technological 

Superiority 

"Technology  is  perhaps  best  understood  as  an  abstract  system  of  knowledge, 
an  attitude  towards  life  and  a  method  for  solving  its  problems"(van  Crevald, 
1989:  1).  It  is  a  competition  with  other  countries  to  develop  technology  to 
gain  advantage. 

Be  Efficient 

Management  practices  known  to  reduce  the  effort  to  develop  technolosv. 

Deter 

Technology  that  discourages  an  adversary  from  acting  because  of  doubt  or 
fear. 

Domestically 

Unduplicated 

Technology  being  developed  commercially  or  expected  to  be  developed 
commercially  should  not  be  built  by  Military  Laboratories. 

Exploit  Adversary's 
Technology  Gaps 

Rapid  advances  in  technology  that  provides  asymmetrical  counters  to  our 
adversary's  military  strengths. 

Feasible 

Technology 

Technology  capable  of  being  developed  within  the  time  horizon  at 
acceptable  cost  with  available  resources  or  with  resources  that  can  be 
developed. 

Maintain 

Technological 

Superiority 

Advances  that  render  portions  or  all  of  an  adversaries  capability  obsolete 
and  prevents  an  adversaries  from  making  our  military  capabilities  obsolete. 

Multiuse 

An  alternative  that  can  support  many  integrated  thrusts  or  projects  is  better 
than  one  that  does  not. 

Prevent 

Exploitation 

Technology  that  prevents  adversaries  from  making  rapid  advances  in 
technology  that  provide  asymmetrical  counters  to  US  military  strengths. 

Simplify 

Operations 

Technology  that  automates  operations  which  simplifies  military  operations 
and  falls  under  the  Simplicity  Principle  of  War. 

Air  Force  People 

-  Talented 

-  Dedicated 

-  Trained 

-  Knowledgeable 

Talented  Air  Force  people  and  contractors  are  the  force  behind  our  ability  to 
win  technological  superiority.  However,  the  question  of  what  is  important 
to  train,  equip,  and  take  care  of  our  people  is  a  separate  question  from  what 
technology  we  should  develop.  The  issue  is  beyond  the  scope  of  this  study. 

Scope  Alternatives 
to  Air  Vehicle 
Technologies 

There  are  many  possible  ways  to  win  technological  superiority,  increase 
awareness,  increase  reach  and  increase  power  for  the  Air  Force.  This  study 
is  limited  to  Air  Vehicles  Options.  This  concept  is  a  constraint. 

Scope  Alternatives 

to  Integrated 
Concepts 

Many  possibilities  exist  to  apply  Air  Vehicles  Technologies.  The  VA 
Director  has  identified  specific  Integrated  Concepts  that  will  be  supported 
by  his  Directorate.  This  sets  direction  and  constrains  technolosv  directions. 

Affordable 

Technology 

The  technology  must  be  with  in  the  budget  of  Air  Vehicles. 

Table  17:  Technology  Leadership  Values  in  USAF  Strategic  Planning 
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Document  or  Concept/Value 

Reach 

Awareness 

Power 

Technology 

Leadership 

Air  Force  Basic  Doctrine 

Unity  of  Command  :  Principle  of  War 

X 

X 

Objective  :  Principle  of  War 

X 

Offensive  :  Principle  of  War 

X 

X 

Mass :  Principle  of  War 

X 

X 

X 

HMj 

X 

X 

X 

IBBaaiaEEHHMl 

X 

X 

Security  :  Principle  of  War 

X 

X 

Surprise  :  Principle  of  War 

X 

X 

X 

Simplicity  :  Principle  of  War 

X 

Global  Engagement 

Rapid  Global  Mobility  ;  Core  Competency 

X 

Precision  Engagement :  Core  Competency 

X 

X 

X 

Global  Attack  :  Core  Competency 

X 

X 

Air  and  Space  Superiority  :  Core  Competency 

X 

X 

X 

Information  Superiority  :  Core  Competency 

X 

Agile  Combat  Support :  Core  Competency 

X 

X 

loint  Vision  2010 

Dominant  Maneuver :  Operational  Concept 

X 

X 

X 

Precision  Engagement :  Operational  Concept 

X 

X 

X 

Full-Dimensional  Protection  :  Operational  Concept 

X 

X 

X 

Focused  Logistics :  Operational  Concept 

X 

X 

Defend  against  asymmetrical  counters  to  US  military  strengths  or 
Technology  Gap  Exploitation 

X 

Maintain  Technological  Superiority 

X 

X 

X 

X 

Incremental  Advancement 

X 

X 

X 

Air  Force  2025 

X 

Reach 

X 

Awareness 

X 

Power 

X 

Air  Force  Strategic  Plan 

Strategic  Agility 

X 

X 

X 

X 

Overseas  Presence 

X 

X 

Power  Projection 

X 

Decisive  Force 

X 

Air  Force  Vision  Statement 

X 

X 

X 

X 

Air  Force  Core  Competencies 

X 

X 

X 

X 

VA  Strategic  Plan  Draft  #  3 

Air  Vehicles  Strategic  Position 

Aircraft  Sustainment 

X 

X 

Space  Superiority/Global  Presence 

X 

X 

UAV  formally  called  "Precision  Strike  Technology” 

X 

X 

X 

Innovative  Core  Technology 

X 

X 

X 

X 

Pre-eminence  in  Research  :  Core  Value 

X 

X 

X 

X 

Talented  and  Dedicated  People 

X 

New  World  Vistas 

Global  Awareness 

X 

Dynamic  Planning  and  Execution  Control 

X 

Mobility 

X 

X 

Projection  of  Lethal  and  Sublethal  Power 

X 

Space  Operations 

X 

X 

People  ' 

X 

Affordability 

X 

Survivability 

X 

Speed 

X 

Range 

X 

Lethality 

X 

Flexibility 

X 

VA  Proposed  Value  Definitions 

X 

X 

X 

Other  Ideas 

Technological  Know-How 

X 

Relevance 

X 

Exploit  Technology  Gaps  of  Adversaries 

X 

Table  18:  Summary  of  Correlations  between  USAF  Strategic  Documents 
and  Reach,  Awareness,  Power,  and  Technological  Superiority 
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Table  19:  Detailed  Qualitative  Correlations  between  USAF  Strategic  Documents  and  Reach,  Awareness,  Power, 

and  Technological  Superiority 
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Documentor  Definition  Reach  Awareness  Power  Technology  Full  Reference 

Concept/Value  Leadership 

Air  and  Space  "Superiority  in  air  and  space  -  control  over  Control  implies  Control  also  "Control  over 

Superiority  :  Core  what  moves  through  air  and  space  -  the  ability  to  implies  what  moves  in 

Comoetencv  delivers  a  fundamental  benefit  to  the  Joint  reach  the  object  knowledge  of  the  air  and 
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Document  or  Definition  Reach  Awareness  Power  Technology  Full  Reference 

Concept/V  alue  Leadership 

Agile  Combat  "Agile  Combat  Support  is  recognized  as  a  The  ability  to  Locations  of 

Support :  Core  core  competency  for  its  central  role  in  reach  enables  current  logistic 

Competency  enabling  air  and  space  power  to  contribute  this  capability.  groups  moving 


lepth,  and  height  of  the  battlespace."  allows  an 
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decision. 
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Air  Force  2025  An  issue  with  Jackson,  Jack  A.,  Brian 
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Document  or  Definition  Reach  Awareness  Power  Technology  Full  Reference 

Concept/Value  Leadership 

Awareness  "Knowledge,  understanding,  or  cognizance  In  Depth  Jackson,  Jack  A.,  Brian 

of  some  thing  or  situation  through  alertness  Definition.  The  L.  Jones,  and  Lee  J. 

in  observing,  detecting,  and  identifying,  so  2025  Analysis  Lehmkuhl.  An 
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Document  or  Definition  Reach  Awareness  Power  Technology  Full  Reference 

Concept/V  alue  Leadership 

Strategic  Agility  "Strategic  agility  is  the  timely  Employment  While  Timely  The  concept  of  (Air  Force  Strategic  Plan 

concentration,  employment,  and  and  sustainment  awareness  is  not  concentration  is  "speed  and  Volume  1  :  The  Future 

sustainment  of  US  military  power  are  concepts  of  specifically  a  method  of  tempo  our  Security  Environment, 
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Document  or  Definition  Reach  Awareness  Power  Technology  Full  Reference 

Concept/V  alue  Leadership 

Power  Projection  .  .power  projection  aims  for  An  in  depth  (Air  Force  Strategic  Plan 

unconstrained  global  reach.  It  is  the  ability  definition  of  the  Volume  1  :  The  Future 

to  deploy  and  sustain  rapidly  and  concept  of  Security  Environment, 
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Document  or  Definition  Reach  Awareness  Power  Technology  Full  Reference 

Concept/Value  Leadership 

Air  Force  Core  "AIR  FORCE  CORE  COMPETENCIES:  Develop  and  Develop  and  Develop  and  Trains  and  (Air  Force  Strategic  Plan 

Competencies  Our  Nation’s  Air  Force  develops,  trains,  sustains  is  sustains  is  sustains  is  integrates  Volume  2  :  Performance 

sustains  and  integrates  the  elements  of  air  captured  in  the  captured  in  the  captured  in  the  elements  of  air  Plan,  Sep  1998  :i) 
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Document  or  Definition  Reach  Awareness  Power  Technology  Full  Reference 

Concept/Value  Leadership 

Innovative  Core  "To  create  and  nurture  innovative  research  If  the  value  If  the  value  If  the  value  "Breakthrough  (Air  Force  Research 

Technology  into  breakthrough  technologies  which  above  is  above  is  above  is  technologies"  is  Laboratory  Air  Vehicles 

enable  revolutionary  warfighter  considered  considered  considered  not  well  Directorate  2010: 
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Concept/Value  Leadership 

Mobility  "Mobility  can  be  the  limiting  factor  in  Mobility  and  its  New  World  New  World  Vistas: 

operations.  Airlift  is  also  in  demand  during  uses  are  a  form  Vistas  defines  Summary  Volume. 
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Projection  of  "The  Air  Force  understands  well  the  issues  Destruction  or  New  World  Vistas: 

Lethal  and  associated  with  projecting  power  from  neutralization  of  Summary J  Volume. 
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satellites  was  addressed.  The  interaction 
between  military  and  commercial  space 
applications  has  not  begun  to  evolve.  It  is 
time,  now,  for  the  Air  Force  to  define  its 
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Nevertheless,  information  is  a  necessary, 
but  not  sufficient  condition  for  victory.  The 
timely  dispersion  of  force  around  the 
globe  and  precision  delivery  of  lethal  or 
non-lethal  weapons  will  remain  an 


essential  contributor 
to  the  final  victory." 
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are  possible. 
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Exploit  Joint  Vision  2010  identifies  the  need  to  An  adversary 

Technology  Gaps  protect  our  forces  against  asymmetrical  cannot  protect 

of  Adversaries  counters  to  our  technology.  An  adversary  every 
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Appendix  B:  Documentation  of  Meetings  with  VA 

VA-AFIT  Value  Hierarchy  Meeting  Highlights/4  Dec  98 

AFRL/VA:  A1  Boudreau,  Lt  Col  Richard  Docken,  Dick  Colclough,  Dr.  David 
Moorehouse,  Jim  Pruner,  Brian  Van  Vliet,  Dr.  Yvette  Weber 
AFIT/ENS:  Capt  Winthrop,  LTC  Kloeber,  Dr.  Deckro 

Subject:  Value  Hierarchy,  1st  two  tiers. 

Comments:  Definitely  support  the  top  level  of  "Support  the  Warfighter" 

Change  Technology  Race  to  Technological  Superiority 

Subject:  2nd  Tier  under  Reach  Awareness  and  Power 
Comments:  Affordability  is  missing  -  Possible  Locations: 

1.  Same  level  as  Gain  and  Sustain 

2.  Below  Gain  for  Reach,  Awareness  and  Power 

3.  Part  of  Technological  Superiority 

General  agreement  that  Gain  and  Sustain  belong  and  are  good  terms  to  use. 

Subject:  Tiers  below  Technological  Superiority 

Comments:  Technology  for  its  own  sake  is  not  included  or  is  not  fully  covered. 

Subject:  Measures  for  Reach 

Comments:  Change  Speed  to  Response  time.  Need  to  include  survivability  in  hierarchy. 


Subject:  Open  Action  Items 


mm 

Capt  Winthrop  will  modify  the  AFRL/VA  model  to  incorporate  affordability. 

AFIT/ENS  and  VA  to  identify  elements  of  “innovation  for  its  own  sake.” 

Done 

AFIT/ENS  to  e-mail  meeting  materials  no  later  than  Monday,  7  Dec  98. 

Dr.  Moorhouse  plans  to  e-mail  briefing  material  of  previous  ROI  work 
through  Mr.  Colclough. 

Capt  Winthrop  will  examine  additional  Air  Force  Strategic  documents  to 
glean  values. 

AFIT/ENS  set  up  meeting  with  VA  Division  Chiefs  to  confirm  values  and 
definitions. 

Done 

AFIT/ENS  set  up  meeting  with  VA  to  confirm  values  and  definitions  ->  1500 
Tuesday,  8  Dec.  ’98. 
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VA-AFIT  Value  Hierarchy  Meeting  Highlights/8  Dec  98 

AFRL/VA:  A1  Boudreau,  Dick  Colclough,  Lt  Col  Richard  Docken,  Jim  Pruner,  Brian 
Van  Vliet,  Dr.  Mike  Zeigler 

AFIT/ENS:  Capt  Winthrop,  LTC  Kloeber,  Dr.  Deckro 

Subject:  Review  of  Reach,  Awareness,  Power,  Technological  Superiority  Definitions 
Comments:  Acceptable  as  written. 

Subject:  Afford  under  Reach,  Awareness,  and  Power  or  under  Gain  and  Sustain 
Comments:  Technically,  Afford  could  be  under  Gain  and  Sustain.  Under  Reach, 
Awareness,  and  Power  because  of  the  political  importance  of  the  value  and  because  it 
allows  calculation  of  lifecycle  cost.  If  broken  out,  this  is  not  possible. 

Subject:  Definition  of  Afford 

Comment:  Afford  needs  to  emphasize  a  reduction  in  cost.  Definition  reworded  for  better 
flow. 

Subject:  Definitions  of  Gain,  Sustain,  and  Afford 

Comments:  There  are  other  ways  of  gaining  reach  besides  technology.  “Through 
technology”  added  to  reflect  that  we  are  considering  technology.  The  order  of  the  three 
values  changed  to  alphabetical  which  also  coincides  with  emphasizing  Afford. 

Subject:  Measures  of  Afford 

Comments:  Afford  can  be  broken  down  into  O&S,  Development  and  Acquisition  cost 
changes  for  the  Air  Force.  However,  the  relationships  are  complex  for  the  three 
measures.  VA  uses  Return  on  Investment  (ROI)  which  is  defined  several  ways  by  VA. 
Attendees  recommend  using  ROI  as  a  composite  measure. 

Subject:  Gain  under  Reach  -  Measures 

Comments:  Changed  from  suggested  into  Response  Time,  Reliability,  Weight,  and  L/D. 
Definitions  need  to  emphasize  the  change  in  the  attribute  pertaining  to  the  system,  not  the 
technology.  Lifetime  is  a  Sustain  measure,  not  Gain. 

Subject:  Sustain  under  Reach  -  Measures 

Comments:  Lifetime  is  more  than  just  hours  added  to  useful  life.  It  also  means  fixing 
existing  systems  to  meet  expected  life.  Number  of  new  capabilities  is  wrong.  It  is  more 
the  quality  of  the  capabilities  being  added.  Mission  flexibility  is  needed  here.  One 
sustainment  program  fixed  an  operational  problem,  decreased  operational  cost,  increased 
the  reliability,  and  resulted  in  small  decreases  in  weight. 


Subject:  Open  Action  Items 


Capt  Winthrop  will  modify  the  AFRL/V A  model  to  incorporate  Gain 

Measures  for  Reach. 

Capt  Winthrop  will  modify  the  AFRL/VA  model  to  incorporate  Sustain 
Measures  for  Reach. 
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AFU/ENS  and  VA  to  identify  elements  of  “innovation  for  its  own  sake.” 

Dr.  Moorhouse  plans  to  e-mail  briefing  material  of  previous  ROI  work 
through  Mr.  Colclough. 

Capt  Winthrop  will  examine  additional  Air  Force  Strategic  documents  to 
glean  values. 

AFIT/ENS  set  up  meeting  with  VA  Division  Chiefs  to  confirm  values  and 
definitions. 

Done 

AFIT/ENS  set  up  follow  on  meeting  with  VA->  0900  Monday,  14  Dec.  ’98. 
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VA-AFIT  Value  Hierarchy  Meeting  Highlights/14  Dec  98 


AFRL/VA:  Dick  Colclough,  Lt  Col  Richard  Docken,  Brian  Van  Vliet,  Yvette  Weber,  Dr. 
Mike  Zeigler 

AFIT/ENS:  Capt  Winthrop,  Dr.  Deckro 

Subject:  Afford  under  Reach,  Awareness,  and  Power 
Comment:  Definition  accepted. 

Subject:  L/D  Measure 
Comment:  Change  L/D  to  AL/D. 

Subject:  Be  Flexible 

Comment:  Removed  from  model  since  it  is  described  by  reliability  and  missions  enabled 
(new  measure).  Missions  enabled  should  be  added  to  both  Gain  and  Sustain  under 
Reach. 

Subject:  Safety 

Comment:  Safety  is  an  issue,  which  is  difficult  to  measure.  Safety  can  be  flight  critical  or 
mission  critical.  Measuring  safety  needs  work.  It  may  be  a  binary  measure. 

Subject:  Measures  under  Awareness  Gain  and  Sustain 

Comment:  VA  believes  the  same  measures  under  Reach  fit  under  Awareness.  The 
problem  is  how  to  deal  with  double  counting.  One  suggestion  was  to  redefine  Reach  to 
look  at  getting  to  the  theatre  while  Awareness  looks  at  in  theatre.  Survivable  and  loiter 
time  are  possible  measures  that  were  suggested.  It  may  be  that  Air  Vehicles  does  not  do 
anything  under  Awareness  and  Power. 

Subject:  Measures  under  Reach 

Comment:  The  measures  and  definitions  for  Gain  and  Sustain  under  Reach  are  acceptable 
now.  These  definitions  are  baseline. 


Subject:  Open  Action  Items 


Done 

Capt  Winthrop  will  modify  the  AFRL/VA  model  to  incorporate  Gain 

Measures  for  Reach. 

Done 

Capt  Winthrop  will  modify  the  AFRL/VA  model  to  incorporate  Sustain 
Measures  for  Reach. 

AFIT/ENS  and  VA  to  identify  elements  of  “innovation  for  its  own  sake.” 

Dr.  Moorhouse  plans  to  e-mail  briefing  material  of  previous  ROI  work 
through  Mr.  Colclough. 

Capt  Winthrop  will  examine  additional  Air  Force  Strategic  documents  to 
glean  values. 

AFIT/ENS  set  up  meeting  with  VA  Division  Chiefs  to  confirm  values  and 
definitions. 

Done 

AFIT/ENS  set  up  follow  on  meeting  with  VA->  0900  Monday,  14  Dec.  ’98. 
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VA-AFIT  Value  Hierarchy  Meeting  Highlights/17  Dec  98 

AFRL/VA:  A1  Boudreau,  Dick  Colclough,  Lt  Col  Richard  Docken,  Dr.  Moorhouse, 
Yvette  Weber 
AFIT/ENS:  Capt  Winthrop 

Subject:  Simplify  Operations 

Comment:  Removed  “automate”  from  the  definition  since  there  are  other  ways 
technology  can  be  used  to  simplify  operations.  It  is  also  possible  to  use  technology  to 
make  operations  more  complex.  Operations  can  be  simplified  by  the  use  of  automation, 
decreasing  people  required  to  perform  an  operation,  or  interoperability. 

Subject:  Mask  and  Prevent  Masking 

Comment:  The  concepts  covered  by  these  words  are  difficult  to  understand  at  first. 
However,  after  explaining  the  concept,  everyone  agreed  the  concepts  made  sense  and 
were  important  to  the  model.  Possible  words  to  replace  mask  are  bypass  and  overcome. 
Examples  of  masking  or  unmasking  also  help.  Examples  include  electronic  warfare, 
using  radar  in  WWD  by  England  to  overcome  air  advantage  by  Germans,  the  threat  of 
nuclear  terrorism  in  a  briefcase  and  using  UAV’s  to  detect  the  weapons,  etc.  These 
concepts  may  be  part  of  the  “Neutralize”  value. 

Subject:  Feasible  is  acceptably  defined 

Subject:  Be  Efficient 

Comment:  Originally  had  “known”  and  “Management”  in  the  definition.  The  practices 
do  not  have  to  be  known  practices  and  management  does  not  fit  in  the  definition  at  all. 

Subject:  Expand  our  Frontier 

Comment:  It  took  some  time  to  get  this  concept  across.  The  concern  is  “Expand  our 
Frontier”  has  connotations  about  evolutionary  technology  and  was  supposed  to  cover 
revolutionary  technology.  The  word  scope  was  confusing  in  the  definition.  Scope  was 
replaced  with  spectrum  where  spectrum  means  an  increase  in  the  technological  base 
allowing  fighting  to  take  place  in  previously  unknown  realms.  Examples  are  the  birth  of 
airpower,  information  dominance,  and  space.  The  definition  was  reworded  somewhat 
and  the  word  revolutionary  was  entered  into  the  definition  to  eliminate  connotations  of 
the  phrase  being  evolutionary,  also. 

Subject:  Deter 

Comment:  Definition  acceptable  as  written  except  that  cost  was  added  to  doubt  and  fear 
as  reasons  to  discourage  an  adversary. 

Subject:  Neutralize 

Comment:  Propose  changing  “obsolete”  to  “ineffective.” 

Subject:  Measures  under  “Feasible” 
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Comment:  “Number  of  Advances”  changed  to  “Number  of  Technological  Challenges.” 
The  definition  was  improved.  Unfortunately,  Technological  Challenges  are  not  all  of  the 
same  difficulty  level.  After  discussion,  it  was  suggested  “Resources  Required.”  The 
definition  for  this  needs  to  be  worked  on  and  checked  with  VA. 

Subject:  Measures  under  “Be  Feasible” 

Comment:  The  measures  suggested  were  acceptable  to  the  group.  The  “Uniqueness” 
measure  can  have  VA  teaming  up  with  more  than  just  industry.  VA  partners  with  other 
Government  agencies,  also.  The  definition  needs  to  be  altered,  slightly. 


Subject: 

Open  Action  Items 

Done 

AFIT/ENS  and  VA  to  identify  elements  of  “innovation  for  its  own  sake.” 

Dr.  Moorhouse  plans  to  e-mail  briefing  material  of  previous  ROI  work 
through  Mr.  Colclough. 

Capt  Winthrop  will  examine  additional  Air  Force  Strategic  documents  to 
glean  values. 

AFIT/ENS  set  up  meeting  with  VA  Division  Chiefs  to  confirm  values  and 
definitions. 

Capt  Winthrop  will  define  “Resources  Required”  under  Feasible 

Done 

Capt  Winthrop  will  modify  definition  of  Uniqueness  to  include  partnering 
with  Gov’t  agencies. 

Done 

Capt  Winthrop  will  work  on  improving  definitions  and  words  for  “mask”  and 
“prevent  masking”  concepts. 

Capt  Winthrop  needs  to  discuss  adding  cost  under  Feasible  as  a  measure  with 
LTC  Kloeber. 

Prepare  for  next  meeting  on  21  Dec  0900.  Send  E-mail  to  those  who  cannot 
attend. 
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VA-AFIT  Value  Hierarchy  Meeting  Highlights/21  Dec  98 

AFRL/VA:  Dick  Colclough,  Lt  Col  Richard  Docken,  Dr.  Moorhouse,  Jim  Pruner,  Rikki 
Peters 

AFIT/ENS:  Capt  Winthrop 

Subject:  Counteract,  Prevent  Counteraction,  Neutralize 

Comment:  The  VA  group  could  not  easily  distinguish  between  the  three  values.  It  is  so 
difficult  to  distinguish  that  VA  recommends  combining  these  concepts  into  one  value. 
Further,  deter  seems  to  fit  as  part  of  these  concepts.  VA  recommends  simplification  of 
the  model. 

Subject:  Simplify  Operations 

Comment:  The  value  is  acceptable  and  the  measures  are  accepted. 

Subject:  Proposed  Measure  for  advantage  level  for  technology 

Comment:  A  possible  measure  measuring  the  advantage  of  technology  on  the  battlefield 
would  have  decapitate,  deter,  neutralize  and  enhance  as  levels.  Decapitate  means  to 
make  an  action  of  an  adversary  unthinkable  or  impossible  because  American  technology 
is  so  powerful.  Deter  would  keep  its  former  meaning  of  technology  that  discourages  an 
adversary  from  acting  because  of  doubt,  fear  or  cost.  Neutralize  would  keep  its  former 
meaning  of  technology  advances  that  render  some  or  all  of  an  adversaries  military 
capability  ineffective  and  cannot  be  countered  at  the  completion  of  the  advance.  Enhance 
is  an  improvement  to  our  forces  technology  giving  us  an  advantage  over  an  adversary. 


Subject:  Open  Action  Items 


Dr.  Moorhouse  plans  to  e-mail  briefing  material  of  previous  ROI  work 
through  Mr.  Colclough. 

Capt  Winthrop  will  examine  additional  Air  Force  Strategic  documents  to 
glean  values. 

AFIT/ENS  set  up  meeting  with  VA  Division  Chiefs  to  confirm  values  and 
definitions. 

OBE 

Capt  Winthrop  will  define  “Resources  Required”  under  Feasible 

Done 

Capt  Winthrop  needs  to  discuss  adding  cost  under  Feasible  as  a  measure  with 
LTC  Kloeber. 

Done 

Capt  Winthrop  to  combine  deter,  neutralize,  counteract,  and  prevent 
counteraction  into  one  value. 

Done 

Capt  Winthrop  to  request  standard  program  proposal  form. 

OBE 

Capt  Winthrop  to  E-mail  latest  briefing  out  for  comments  over  Christmas 

Done 

Prepare  for  meeting  in  first  week  in  January. 
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VA-AFIT  Value  Hierarchy  Meeting  Highlights/7  Jan  99 

AFRL/VA:  Lt  Col  Richard  Docken,  Mike  Ziegler 
AFIT/ENS:  Capt  Winthrop,  Dr.  Dick  Deckro,  LTC  Jack  Kloeber 

Subject:  Reusability  and  recoverability  measures  under  Power  Gain  and  Sustain 
Comments:  Reusability  and  recoverability  appear  similar.  The  difference  is  reusability  is 
time  before  the  entire  system  must  be  overhauled  and  recoverability  is  the  turn  around 
time  in  between  missions.  Suggestions  for  measures  are  turn  around  time,  mission 
capable  rate,  and  sortie  generation  rate. 

Subject:  Detectability,  countermeasures  and  vulnerability  under  survive  under 
Power/Gain  and  Sustain 

Comments:  These  concepts  are  essentially  right.  However,  the  wrong  words  are  being 
used.  Better  words  are  vulnerability,  susceptibility,  and  detectability.  Participants 
recommended  talking  with  SURVIAC  to  find  documents  that  discuss  these  concepts. 

Subject:  Add  Lethality  to  Power  under  Gain  and  Sustain 

Comments:  Lethality  may  not  be  what  VA  does,  but  should  be  added  as  an  opener  for 
future  discussion. 

Subject:  Changes  in  Technological  Superiority  are  acceptable 

Comments:  The  word  adversary  is  used  a  lot  in  the  model.  Adversary  needs  definition. 
This  should  not  be  a  large  problem  since  VA  has  documentation  that  tells  us  who  the 
adversary  is.  Another  issue  here  is  the  type  of  war  being  fought.  Nuclear  weapons  are 
not  overwhelming  in  all  possible  types  of  war  (conventional,  nuclear,  irregular). 

Subject:  Advantage  Categories 

Comments:  The  categories  make  sense  as  written.  This  category  is  likely  to  be  a  source  of 
controversy  during  scoring. 

Subject:  Expand  our  Frontier  and  Simplify  Operations 

Comments:  These  definitions  overlap  somewhat.  Specifically,  the  concept  of  enabling 
technology  that  simplifies  operations  and  “Potential  for  New  warfighting  capability”  are 
difficult  to  tell  apart  in  some  situations. 


Subject:  Open  Action  Items 


OBE 

Dr.  Moorhouse  plans  to  e-mail  briefing  material  of  previous  ROI  work 
through  Mr.  Colclough. 

Capt  Winthrop  will  examine  additional  Air  Force  Strategic  documents  to 
glean  values. 

AFIT/ENS  set  up  meeting  with  VA  Division  Chiefs  to  confirm  values  and 
definitions. 

Done 

Clarify  reusability  and  recoverable  definitions 

Change  definitions  and  names  for  Detectability,  countermeasures  and 
vulnerability  under  survive  under  Power/Gain  and  Sustain 
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Add  Lethality  under  Power  both  Gain  and  Sustain  and  define  measure 

Define  Adversary  when  evaluation  measures  are  being  built 

Remove  overlap  in  definitions  or  evaluation  measures  for  Expand  our 

Frontier  (Potential  for  new  warfighting  capability)  and  Simplify  Operations 
(enable) 
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VA-AFIT  Value  Hierarchy  Meeting  Highlights/19  Jan  99 

AFRL/VA:  Dick  Colclough,  A1  Boudreau 

AFIT/ENS:  Capt  Winthrop,  Dr.  Dick  Deckro,  LTC  Jack  Kloeber 

Subject:  Breakout  Space  and  Air  evaluation  Measures  Question 
Comments:  AFIT  brought  up  concerns  about  differences  in  value  for  space  and  air 
evaluation  measures.  The  question  was  whether  weight  reduction  should  be  different  for 
space  programs  and  air  programs  since  it  is  weighted  differently  for  space  vehicles  then 
for  air  vehicles  and  is  scored  differently.  That  is,  a  savings  for  200  pounds  on  a  space 
vehicle  can  have  much  higher  value  than  200  pounds  saved  on  an  aircraft.  This  concept 
was  rejected.  Air  Vehicles  felt  the  value  model  should  be  equally  applicable  to  both 
types  of  vehicles  and  no  special  weightings  should  be  given  due  to  the  type  of  vehicle 
under  consideration. 
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VA-AFIT  Value  Hierarchy  Meeting  Highlights/20  Jan  99 

AFRL/VA:  Dick  Colclough,  Dr.  Moorhouse,  Mike  Ziegler,  Brian  Van  Vliet 
AFIT/ENS:  Capt  Winthrop,  Dr.  Dick  Deckro,  LTC  Jack  Kloeber 

Subject:  Lethality 

Comments:  Concepts  of  payload  and  energy  generated  made  no  sense  here.  These 
measures  were  replaced  with  effectiveness  and  quantity.  Air  Vehicles  directly  influences 
these  measures. 

Subject:  Regeneration  and  Reusability 

Comments:  VA  cannot  distinguish  between  reliability  and  reusability.  The  two  metrics 
should  be  combined  under  an  availability  measure.  Availability  should  be  placed  under 
Reach,  not  power.  Additionally,  regeneration  should  be  placed  under  reach.  Availability 
can  be  defined  from  Air  Force  documentation.  Regenerate  should  be  redefined  in  terms 
without  using  “generate”  in  it. 

Subject:  Feasible 

Comments:  VA  renamed  number  of  technology  challenges  to  technological  challenge. 
This  measure  will  be  subjective  (High,  Medium,  Low).  There  is  no  known  way  to 
improve  this  measurement.  The  concept  of  TRL  was  proposed  as  a  possible  measure. 
This  measure  is  about  maturity  level  and  is  covered  by  the  concept  of  technological 
challenge. 

Subject:  Warfare  advantage 

Comments:  VA  cannot  distinguish  between  different  types  of  warfare  and  the  advantage 
level  (i.e.  nuclear  war,  conventional  war,  irregular  war,  etc.).  The  technology  they 
develop  is  not  developed  with  these  concepts  in  mind. 

Subject:  Cost  to  adversary 

Comments:  This  cannot  be  estimated  as  a  percent  of  GNP  for  another  country.  At  best, 
this  either  is  a  subjective  measure  or  can  be  measured  in  orders  of  magnitude.  A  project 
that  costs  more  orders  of  magnitude  than  another  for  our  closest  technical  adversary  to 
overcome  would  be  better  for  us  then  one  that  costs  less. 

Subject:  “Closest  Adversary” 

Comments:  Definition  of  our  closest  adversary  makes  no  sense.  This  term  was  changed 
slightly  to  be  our  closest  technical  adversary.  The  term  then  makes  sense  to  VA. 


Subject:  Open  Action  Items 


Capt  Winthrop  will  examine  additional  Air  Force  Strategic  documents  to 
glean  values. 

Done 

AFIT/ENS  set  up  meeting  with  VA  Division  Chiefs  to  confirm  values  and 
definitions. 

Done 

Remove  overlap  in  definitions  or  evaluation  measures  for  Expand  our 

Frontier  (Potential  for  new  warfighting  capability)  and  Simplify  Operations 
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(enable) 

Prepare  for  1  Feb  meeting  with  Col  Wood. 

Prepare  for  29  Feb  meeting  on  evaluation  measures  with  Integrated  Concept 
Leads  and  Strategic  IPT  leads. 

Done 

Set  up  a  meeting(s)  with  VAS,  VAA,  and  or  VAC  to  develop  some  evaluation 
measures. 
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VAA:  L/D  Evaluation  Measure/25  Jan  99 


Anderson  identifies  L/D  or  lift-to-drag  ratio  as  a  measure  of  the  aerodynamic 
efficiency  of  an  airplane  (Anderson,  1985:  256).  For  further  information  on  lift  and  drag 
concepts,  consult  Anderson  or  any  other  standard  aeronautical  engineering  text.  The 
discussion  below  is  a  summary  of  an  interview  with  Mr.  Russ  Osbourne  of  the  Air 
Vehicles  Directorate  at  AFRL  held  on  25  Jan  1999. 

L/D  depends  on  what  air  vehicle  type  is  being  discussed  as  shown  in  Table  20. 
Therefore,  this  quantity  must  be  considered  as  the  ratio:  (L/D)n /(L/ D)0 .  The  difference 

(L/D)n  -(L/D)q  was  considered  as  the  X-axis  value,  but  it  inaccurately  describes  reality. 

Specifically,  a  change  in  L/D  from  2  to  4  is  not  the  same  as  the  change  from  4  to  6. 
However,  a  change  of  L/D  from  2  to  4  is  the  same  as  a  change  of  20  to  40  in  terms  of 
value  to  Air  Vehicles. 


L/D 

Vehicle  Type 

E39I 

Hypersonic  Cruise  Vehicles  and  some  supersonic  vehicles  (F-15) 

mm 

Transonic  Fighters 

Supersonic  Transports 

wmm 

Bombers 

15-28 

Transports  (C-5,  C-141,  etc.) 

28-40 

Sail  Planes  and  Endurance  Vehicles 

Table  20 

:  L/D  categories  for  Aerospace  Vehicles  (Source:  Persona 

Interview  with  Mr.  Russ  Osborne  in  Air  Vehicles  Directorate  of  AFRL) 

The  highest  value  for  (L/D)n/(L/D)0  was  set  at  2  and  the  lowest  value  was  set  at 

2/3.  This  considers  conventional  air  vehicles  and  possible  future  developments  in 
hypersonic  vehicles.  Doubling  L/D  is  unlikely  in  most  situations,  although  endurance 
vehicles  and  hypersonic  vehicles  might  achieve  this.  The  lower  value  of  2/3  was  set  with 
the  idea  that  L/D  might  be  traded  off  against  other  attributes  resulting  in  a  reduction  of 
L/D  for  a  system  with  corresponding  increases  in  other  attributes.  There  is  little  chance 
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of  (L/ D)n  /(L/ D)0  being  allowed  to  drop  any  less  than  2/3  in  a  real  situation.  An 

example  of  this  is  when  weapon  pods  or  instrumentation  pods  are  placed  on  fighters. 
These  pods  could  increase  the  power  or  awareness  of  the  weapon  system  but  will 
decrease  L/D. 

The  process  defined  in  Chapter  2,  Error!  Reference  source  not  found.,  was 
implemented  for  a  continuous  evaluation  measure.  It  was  determined  that  value  versus 
(L/ D)n  /(Li D)0  was  a  straight  line.  A  drop  in  L/D  from  1  to  anything  less  than  1  should 

be  penalized  and  follows  the  same  straight  line  value  function  as  increasing  the  L/D  does. 
Hence,  this  function  requires  rescaling  so  the  value  will  be  positive.  Figure  23  shows  the 
both  the  transformed  and  original  value  functions  developed. 


Figure  23:  (L/D)n/(L/D)0  Value  Function 
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VAC-AFIT  Reach  Measures/27  Jan  99 


AFRL/VAC:  Dennis  Sedlock,  Ralph  Speelman,  Bruce  Clough,  Roy  Robertas, 
AFIT/ENS:  Capt  Winthrop,  Dr.  Dick  Deckro 

Subject:  Regeneration 

Comment:  Regeneration  concept  might  be  better  if  changed  to  turn-around-time  or  down 
time. 

Subject:  Availability 

Comment:  VAC  has  expertise  on  the  reliability  portion  of  this  concept. 

Subject:  Evaluation  Measure  expertise 

Comment:  VAC  believes  they  have  expertise  to  help  develop  Availability,  Missions 
Enabled,  Regeneration  and  Safety  Measures.  Logistical  expertise  is  required  for  some  of 
these  measures. 


Subject:  Open  Action  Items 


Done 

Capt  Winthrop  will  examine  additional  Air  Force  Strategic  documents  to 
glean  values. 

Done 

Verify  no  overlap  in  definitions  or  evaluation  measures  for  Expand  our 

Frontier  (Potential  for  new  warfighting  capability)  and  Simplify  Operations 
(enable). 

Done 

Prepare  for  1  Feb  meeting  with  Col  Wood. 

Done 

Prepare  for  29  Feb  meeting  on  evaluation  measures  with  Integrated  Concept 
Leads  and  Strategic  IPT  leads. 

ISEiSIl 

Prepare  for  2  Feb  meeting  with  VAC. 

Contact  Bill  Baron  and  Doug  Dolvin  of  VAS  for  weight  reduction  measure. 

158 


VA-AFIT  Reach,  Power,  Tech.  Sup.  Measures/29  Jan  99 

AFRL/VA:  Dick  Colclough,  Brian  Van  Vliet,  A1  Boudreau,  Yvette  Weber,  Rikki  Peters 
AFIT/ENS:  Capt  Winthrop,  Dr.  Dick  Deckro,  LTC  Jack  Kloeber 

Subject:  Evaluation  Measure  Ranges 

Comments:  Ranges  for  most  of  the  evaluation  measures  were  developed.  The 
information  is  documented  in  Appendix  F:  Air  Vehicles’  Evaluation  Measures  on  page 
199.  Some  important  elicitation  results  are  documented  below. 

Subject:  Gain  and  Sustain  under  Power 

Comments:  VA  cannot  differentiate  between  gain  and  sustain  issues  under  Power.  Gain 
and  sustain  values  were  removed  from  the  Power  value.  Lethality,  afford  and  survival 
are  the  important  values  for  power  for  Air  Vehicles. 

Subject:  Gain  and  Sustain  definitions 

Comments:  VA  thinks  of  sustain  as  increasing  the  life  of  existing  weapon  systems  while 
gain  is  development  of  new  weapon  systems.  Gain  and  sustain  definitions  require  some 
changes. 

Subject:  AF  Savings  Ratio  Evaluation  Measure 

Comments:  Space  programs  have  the  potential  of  savings  of  100  for  this  measure.  Air 
vehicles  are  not  likely  to  gain  more  than  1.2  or  1 .5  for  the  measure.  The  savings  to  the 
Air  Force  is  calculated  using  recurring  costs  and  nonrecurring  costs.  Savings  are  fairly 
easy  to  estimate.  It  will  also  include  leveraging  (cost  savings  multiplied  by  number  of 
weapon  systems).  The  Air  Vehicles’  investment  cost  is  difficult  to  estimate. 

Subject:  L/D 

Comments:  The  measure  needs  to  be  revisited.  Space  vehicles  using  hypersonic 
technology  have  the  potential  tripling  L/D  from  existing  systems  (space  shuttle). 

Subject:  Response  Time 

Comments:  Goals  are  to  be  anywhere  in  the  world  within  3  hours.  It  assumes  travel 
distance  of  12,000  miles.  Current  time  is  24  hours.  This  measure  is  for  an  individual 
weapon  system. 

Subject:  Multiuse 

Comments:  Discussions  on  multiuse  brought  up  the  need  to  determine  the  rating  level 
each  MAJCOM  places  on  an  VA  program.  All  VA  customers  are  MAJCOMs,  who  must 
be  satisfied.  Multiuse  at  one  time  considered  supporting  both  integrated  concepts  in  VA 
and  supporting  outside  users.  The  discussion  defined  outside  users  as  USAF  MAJCOMs 
and  established  the  new  measure. 

Subject:  Future  Opportunities  and  New  Warfighting  Spectrum  under  Expand  our 
Frontier 
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Comments:  It  is  the  experience  of  the  group  that  human  beings  have  nearly  always 
underestimated  the  potential  of  technology.  As  such,  both  measures  are  binary.  Further, 
Future  Opportunities  is  the  wrong  concept  and  has  been  redefined  as  revolutionary. 
Revolutionary  is  defined  as  a  scientific  advance  that  could  potentially  change  VA’s  entire 
paradigm  and  learning  curve. 

Subject:  Feasible  Value  under  Technological  Superiority 

Comments:  The  evaluation  measure  concepts  of  technology  challenge  and  readiness  time 
do  not  fit.  Technology  challenge  is  not  the  right  concept.  A  better  concept  is 
technological  risk.  This  will  be  defined  in  terms  of  the  unknowns  faced.  The  concept  of 
readiness  time  will  be  removed  from  the  value  model  and  will  be  treated  as  a  constraint. 
VA  is  generally  given  deadlines  for  maturing  technology  to  a  useable  level.  The  current 
maturity  level  is  measured  by  the  VA  technology  readiness  scale  measurement. 
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VA  Directorate-AFIT  Meeting  Highlights/1  Feb  99 

AFRL/VA:  Dr.  Borger 

AFIT/ENS:  Capt  Winthrop,  Dr.  Dick  Deckro,  LTC  Jack  Kloeber 
Subject:  Approval  of  Value  Model 

Comments:  The  value  model  takes  into  account  Air  Force  Strategic  values,  which  is 
critical.  Dr.  Borger  recommends  changing  the  problem  statement  slightly  to  say  “be 
consistent  with  Air  Force  values”  rather  then  “its  values,”  which  implies  Air  Vehicles 
values  not  Air  Force  values. 

Dr.  Borger  explained  how  aeronautics  is  almost  entering  its  100th  year  (since  1903 
with  the  Wright  brothers)  and  is  wondering  if  everything  that  can  be  done  has  been  done 
in  the  field.  The  measures  in  the  value  hierarchy  are  traditional  measures.  He  wonders 
what  measures  we  should  be  using  now?  Examples  of  industries  with  lots  of  measures 
they  are  working  with  are  the  pharmaceutical  industry  and  electronics  industry.  These 
industries  invest  10%-15%  of  their  profits  in  R&D.  The  oil  industry  invests  1%  of  their 
profits  in  R&D.  Is  aeronautics  like  the  oil  industry  now  or  can  it  be  like  the 
pharmaceutical  or  electronics  industries? 
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VAS-AFIT  Weight  Reduction  Measure/2  Feb  99 

AFRL/VA:  VAS/William  Baron,  Douglas  Dolvin  (Had  to  leave  early) 

AFIT/ENS:  Capt  Winthrop 

The  X-axis  for  this  evaluation  measure  was  defined  as  the  ratio: 

Airframe  Weight  J  /[  Airframe  Weight  1  ^  the  ratio_  ..N>.  is  slands  for  using 

Dry  Vehicle  Weight \N  /  [Dry  Vehicle  Weight  J0 

the  technology  in  a  new  system  while  “0”  stands  for  the  baseline  system  being  compared 

with.  A  typical  value  for  Airframe  Weight  -n  current  fighters  is  0.35.  The 

[Dry  Vehicle  Weight  J0 


Airframe  Weight 
Dry  Vehicle  Weight 


might  ever  become  based  on  current  understanding  and  pushing 

N 


to  the  limits  is  0.22.  This  ratio  always  assumes  a  particular  mission  has  been  selected, 
but  by  dividing  the  new  system  by  the  old  system,  the  mission  parameter  is  canceled  out. 
The  value  function  was  determined  to  be  a  straight-line  curve  for  this  evaluation  measure. 

While  selecting  a  mission  sets  the  likely  values  for  the  ratio,  selecting  the  mission 
can  be  difficult.  For  existing  missions,  it  is  clear  existing  weapon  systems  should  be 
examined.  It  is  important  to  pick  the  best  known  technology  for  the  mission  class  for  this 
evaluation  measure  to  work.  It  might  be  possible  to  “game  the  system”  by  comparing 
new  technology  with  something  less  than  deployed  state  of  the  art.  The  effect  would  be 


to  inflate  this  evaluation  measure. 


Another  issue  with  the  evaluation  measure  occurs  when  evaluating  a  technology 


for  a  mission  that  has  never  been  performed.  In  this  situation,  the  evaluators  should 
identify  a  baseline  system  that  enables  the  mission  to  be  performed.  Then,  alternatives 


can  be  proposed  that  improve  over  the  baseline  system. 
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Weight  reduction  technologies  will  consists  of  likely  consist  of  several 
technologies  which  together  may  reduce  weight  more  than  the  sum  of  the  individual 
technologies.  That  is,  technology  T1  by  itself  might  reduce  weight  in  a  system  by  1  % 
and  technology  T2  might  reduce  weight  by  2  %.  However,  combining  technologies  T1 
and  T2  in  the  system  might  reduce  the  weight  by  5  %.  Hence,  an  alternative  technology 
should  include  several  technologies  working  together. 

Considerable  uncertainty  may  result  in  the  measure  of  merit  since  it  is  uncertain 
how  combing  technologies  will  actually  reduce  weight.  In  the  situation,  the  technological 
risk  becomes  very  important  since  it  will  help  establish  the  likelihood  of  achieving  a 
particular  desired  result. 


Additional  information  on  Airframe  Weight 

_  Dry  Vehicle  Weight  J 

day  or  two.  This  may  change  the  X-axis  scale,  somewhat 


is  expected  within  the  next 
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VAC-AFIT  Reach  Measures/2  Feb  99 


AFRL/VAC:  Maj  Jeff  Cole,  Ralph  Speelman,  Bruce  Clough,  Mart  Lentz 
AFIT/ENS:  Capt  Winthrop,  Dr.  Dick  Deckro 

Subject:  Awareness 

Comments:  VAC  works  on  technologies  that  improve  awareness.  An  example  is  health 
monitoring  systems,  which  potentially  could  be  used  to  reduce  the  time  required  to 
diagnose  problems  in  aircraft.  Another  possible  way  of  looking  at  this  would  be  to 
identify  the  number  of  systems  a  health  monitoring  system  can  monitor.  It  is  also 
possible  that  the  health  system  could  determine  what  actually  caused  combat  damage  and 
could  use  this  information  as  a  source  of  intelligence  about  an  adversary. 

Subject:  L/D  and  Weight  Reduction 

Comments:  These  measures  seem  out  of  place  in  the  hierarchy  in  comparison  to  the  other 
measures  in  the  same  group.  A  possible  fix  is  to  move  these  two  measures  down  a  level 
in  the  hierarchy  and  place  them  under  a  higher  tier  value  such  as  “Mission 
Effectiveness.” 

Subject:  Loss  of  an  Aircraft 

Comments:  The  cost  of  the  loss  of  an  aircraft  is  probably  not  well  accounted  for  in  the 
model.  Cost  of  an  aircraft  loss  could  be  scored  in  the  savings  portion  (lack  of  savings)  to 
the  Air  Force. 

Subject:  Gain  Availability 

Comments:  Availability  needs  to  include  “a  single  system  which  prevent  the  Aircraft 
from  performing  its  mission”  in  the  definition.  Modern  aircraft  are  typically  available 
80%-90%  of  the  time.  The  measure  is  PAN/PA0  ,  which  can  range  from  0  to  1.5.  This 

allows  for  an  increase  up  to  100%  in  most  situations  and  for  decreasing  availability  in  the 
case  of  a  new  weapon  system  performing  a  new  mission.  The  measure  assumes  time 
between  failure  and  time  to  repair  is  considered  in  determining  the  probability  of 
availability.  The  shape  of  the  value  function  is  an  s-curve  which  will  require  verification 
on  the  steepness.  This  attribute  is  mutually  exclusive  with  Regeneration  because  it  does 
not  cover  repair  of  battle  damage  or  the  time  to  replenish  consumeables  on  the  weapon 
system. 

Subject:  Sustain  Availability 

Comments:  The  same  changes  to  definition  and  the  assumptions  mentioned  in  “Gain 
Availability”  apply  here.  In  Sustainment,  there  is  no  value  for  decreasing  the  availability 
and  a  program  would  not  do  this.  The  curve  is  the  same  as  the  Gain  Availability  curve 
when  the  measure  is  greater  than  1. 

Subject:  Gain  Missions  Enabled  and  Sustain  Missions  Enabled 
Comments:  Adding  missions  to  a  weapon  system  takes  place  fairly  often.  In  no  situation 
will  a  mission  be  taken  away  from  an  aircraft  and  without  adding  a  new  mission,  so  the 
lowest  this  measure  can  be  is  0.  It  is  common  to  add  a  single  mission  to  a  weapon  system 
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and  possible  though  unusual  to  add  2  missions  to  an  aircraft.  Most  of  the  value  for  the 
function  in  this  measure  is  gained  by  adding  a  single  new  mission  and  adding  two 
missions  is  “icing  on  the  cake.” 

Subject:  Regeneration 

Comments:  The  evaluation  measure  should  be  redefined  to  encompass  the  turn  around 
time  on  the  ramp  rather  than  defining  it  in  terms  of  sorties.  The  operations  should  be 
“normal  mission”  operations,  rather  then  “normal  operations.”  Scoring  this  attribute 
requires  consideration  of  combat  damage  and  replenishment  of  consumables.  Repair  of 
combat  damage  includes  assessment,  design  of  repair,  repair  execution  and  accreditation. 
Replenishment  of  consumables  includes  rearming,  refueling,  and  downloading  mission 
specific  information.  The  evaluation  measure  is  mutually  exclusive  with  Availability 
because  it  does  not  include  reliability  or  the  time  to  repair  normal  failures  of  equipment. 

Subject:  Safety 

Comments:  Safety  is  to  be  redefined  as  the  change  in  the  number  of  class  A  incidents  that 
take  place  or  ARo.  The  number  of  incidents  is  typically  measured  per  hour.  A  doubling 
in  the  number  of  safety  incidents  is  bad  and  has  the  least  value.  Decreasing  the  number 
of  incidents  to  0  is  the  best  case  scenario. 
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VA-AFIT  Reach,  Power,  Tech.  Sup.  Measures/4  Feb  99 

AFRL/VA:  Dick  Colclough,  Mike  Ziegler,  Brian  Van  Vliet,  A1  Boudreau,  Yvette  Weber 
AFIT/ENS:  Capt  Winthrop,  LTC  Jack  Kloeber 

Subject:  VAC  Question  -  Lifecycle  cost  not  calculated  in  all  programs 
Comments:  The  Strategic  IPT  and  Integrated  Concept  Leaders  recognize  this  issue.  They 
believe  it  will  be  necessary  to  compute  this  quantity  in  the  future.  It  is  expected  that 
calculation  of  AF  Savings  Ratio  will  be  difficult  for  the  engineers  of  VA.  This  quantity 
should  be  watched  closely  and  if  a  better  measure  is  found,  it  may  be  replaced. 

Subject:  VAC  Question  -  Loss  of  aircraft  due  to  attrition  not  explicitly  calculated  in 
model 

Comments:  Air  Force  regulations  do  not  permit  this  quantity  to  be  used. 

Subject:  The  6.1, 6.2,  and  6.3  customer  is  not  considered  in  the  model 

Comments:  It  is  true  these  groups  are  customers.  However,  the  ultimate  customer  is  the 

warfighter.  Another  customer  is  the  System  Program  Offices  (SPO’s).  The  evaluation 

measure  called  “User  Rating”  was  modified  to  allow  consideration  of  endorsements  by 

SPO’s. 

Subject:  Alternatives 

Comments:  Air  Vehicles  plans  to  provide  2  each  of  their  red,  green  and  blue  programs 
for  grading  by  the  model.  Further,  they  will  provide  names  of  contacts  of  who  can  score 
these  programs.  A  red  program  is  one  on  the  critical  path  for  their  roadmap.  A  green 
program  is  an  important  program,  but  not  critical.  A  blue  program  is  a  “nice  to  have” 
program  and  is  the  first  to  go  in  case  of  budget  cuts. 

Subject:  Afford 

Comments:  AFIT  realized  a  problem  existed  in  dividing  affordability  between  Reach  and 
Power.  How  much  money  is  Reach  savings  and  how  much  is  Power  savings?  VA 
agreed  there  was  a  problem  and  Afford  was  moved  under  Technological  Superiority  so 
all  savings  could  be  measured  in  one  place. 

Subject:  Evaluation  Measures 

Comments:  Nearly  all  evaluation  measures  were  determined  in  the  meeting.  Quantity 
under  Lethality  was  changed  to  Firepower.  No  definition  for  Firepower  was  available  in 
the  meeting.  Both  Firepower  and  Warfare  Advantage  measures  were  deferred  to  the  Air 
Vehicles’  Directorate  level.  The  Ratio  of  People  is  believed  to  be  difficult  to  measure  for 
a  technology.  A  technologist  might  have  better  luck  with  part  counts.  Program  managers 
in  VA  may  have  better  ideas  for  this  quantity. 
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VA  Directorate-AFIT  Weight  Elicitation  Summary/4  Feb  99 

AFRL/VA:  Dr.  Borger 

AFIT/ENS:  Capt  Winthrop,  Dr.  Dick  Deckro 

Subject:  Awareness 

Comments:  The  decision-maker  believes  Air  Vehicles  will  contribute  to  Awareness  in  the 
future  in  an  integration  role.  He  believes  availability,  missions  enabled,  and  regeneration 
are  the  measures  of  the  value.  These  measures  are  expected  to  be  the  same  as  the  values 
under  Reach/Gain  and  are  expected  to  score  the  same  as  those  under  Reach/Gain.  VA 
has  no  sustainment  value  in  Awareness,  but  this  may  change  in  the  future. 

Subject:  Rank  Order  of  Reach  Evaluation  Measures 

Comments:  The  decision-maker  ranked  ordered  the  evaluations  measures  as  follows:  1) 
Availability,  2)  Response  Time,  3)  Missions  Enabled,  4)  Regeneration  and  5)  Mission 
Effective  for  Gain  under  Reach.  The  decision-maker  ranked  ordered  the  evaluation 
measures  as  follows:  1)  Achieve/Extend  Life,  2)  Missions  Enabled,  3)  Availability,  4) 
Regeneration,  5)  Safety,  6)  Weight  Reduction  for  Sustain  under  Reach. 

Subject:  Power  Evaluation  Measures  Ranking 

Comments:  During  the  elicitation,  the  decision-maker  realized  Standoff  was  important  as 
a  measure  of  power.  The  decision-maker  ranked  evaluation  measures  as  follows:  1) 
Effectiveness,  2)  Firepower,  and  3)  Survival.  However,  when  asked  to  compare 
Effectiveness  with  Achieve/Extend  Life,  he  realized  Standoff  was  missing  from  Power. 
Adding  Standoff  changed  the  weighting  he  provided  between  Power  and  Reach.  The 
new  ranking  was:  1)  Standoff,  2)  Effectiveness,  3)  Firepower,  and  4)  Survival. 

Subject:  Technological  Superiority  measures  Ranking 

Comments:  The  decision-maker  ranked  Be  Efficient  Measures  as  follows:  1)  User 
Rating,  2)  Multiuse,  3)  Uniqueness,  4)  Dual  Use,  5)  Intermediate  Projects.  The  decision¬ 
maker  ranked  Expand  our  Frontier  measures  as  follows:  1)  New  Warfighting  Spectrum, 

2)  Revolutionary. .  The  decision-maker  ranked  Simplify  Operations  measures  as  follows: 
1)  Enables,  2)  Interoperability,  3)  Ratio  of  People.  The  decision-maker  ranked  Win 
measures  as  follows:  1)  Warfare  Advantage,  2)  Lead,  3)  Cost  to  Adversary. 

Subject:  Reach/Gain  versus  Sustain 

Comments:  The  decision-maker  commented  that  sustainment  in  Air  Vehicles  will 
become  less  important  and  Gain  more  important  around  2002. 

Subject:  Weights  of  Values 

Comments:  Summary  of  elicited  weights  follows.  Note:  The  is  used  interchangeably 
with 


Weight  Variable 

Definition 

W  Aware 

Awareness 

WA-Ga 

Awareness/Gain 
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Weight  Variable 

Definition 

WAG-Av 

Awareness/Gain/Availability 

W  Power 

Power 

Wp-Le 

Power/Lethality 

WpL-St 

Power/Lethality/Standoff 

WReach 

Reach 

WR-Ga 

Reach/Gain 

WrG-Av 

Reach/Gain/Availability 

Wr-Su 

Reach/Sustain 

Reach/Sustain/Achieve/Extend  Life 

Technological  Superiority 

Wj-AF 

Technological  Superiority/ Afford 

Wta-AF 

Technological  Superiority/ Afford/ AF  Savings  Ratio 

Wj-Be 

Technological  Superiority/Be  Efficient 

WTB-Us 

Technological  Superiority/Be  Efficient/User  Rating 

Wj-Ex 

Technological  Superiority/Expand  our  Frontier 

WjE-Ne 

Technological  Superiority/Expand  our  Frontier/New  Warfighting 

Spectrum 

Wr-PR 

Technological  Superiority/Programmatic  Risk 

Wt-SO 

Technological  Superiority/Simplify  Operations 

WiSO-En 

Technological  Superiority/Simplify  Operations/Enables 

Wj-Wi 

Technological  Superiority/Win 

WjE-Wa 

Technological  Superiority/Win/W arfare  Advantage 

Elicited  Weights  from  Air  Vehicles'  Head  Office. 


1 .  Purpose:  Relate  Reach  to  Awareness 

Reach/Sustain/ Achieve/Extend  Life  is  three  times  as  important  as 
Awareness/Gain/ Availability. 

w  RS_Acw  R_Su'w  Reach=^'  (w  AG_Av'w  A_Ga'w  Aware) 
w  A  Ga=  1 


2.  Purpose:  Relate  Reach  to  Power 
Achieve/Extend  Life  is  as  important  as  standoff 

w  RS_Ac  w  R_Su  w  Reach=w  PL_Sfw  P_Le'w  Power 

w  P_Le= 1 


3.  Purpose:  Relate  Tech.  Sup.  to  Reach 

New  Warfighting  Spectrum  is  twice  as  important  as  Sustain/Extend  Life. 
2'  (w  RS_Ac  w  R_Suw  Reach) =w  TS'W  T_Wi" w  TE_Wa 


4.  Purpose:  Relate  Reach/Gain  to  Reach/Sustain 
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Reach/Sustain/Achieve/Extend  Life  is  twice  as  important  as  Reach/Gain/Availability. 
Sustainment  will  be  most  important  until  2002  when  gain  will  increase  in  importance. 

w  RS_Acw  R_Suw  Reach*2-  (w  RG_Avw  R_Ga-w  Reach) 

Reduces  to: 

w  RS_Ac-w  R_Su*2-(w  RG_Avw  R_Ga) 

5.  Purpose:  Relate  Tech.  Sup./Be  Eff.  to  Tech.  SupVExp.  our  Fron. 

User  Rating  is  as  important  as  New  Warfighting  Spectrum 

w  TSW  T_Be  w  TB_Us*w  TSW  TJEx  w  TE_Ne 
Reduces  to: 

w  T_Be'w  TB_Us*w  T_Ex'w  TE_Ne 


6.  Purpose:  Relate  Tech.  Sup./Be  Eff.  to  Tech.  Sup./Simplify  Operations 
User  Rating  is  twice  as  important  as  Enables 

w  TBew  tb_Us=2-(w  T_SO-w  TSO_En) 

7.  Purpose:  Relate  Tech.  SupVExp.  our  Fron.  to  Tech.  SupVWin 
Warfare  Advantage  is  as  important  as  New  Warfighting  Spectrum. 

w  T_Exw  TE_Ne*w  T_Wiw  TE_Wa 

8.  Purpose:  Relate  Tech.  Sup./Exp.  our  Fron.  to  Tech.  Sup./Win 
Warfare  Advantage  is  three  times  more  important  then  AF  savings  Ratio. 

w  T_Wi- w  TE_Wa*3-  (w  T_AF-W  TA_Af) 

w  TA_AF* 1 

9.  Purpose:  Relate  AF  Savings  Ratio  to  Programmatic  Risk 
AF  Savings  Ratio  is  5  times  as  important  as  Programmatic  Risk 
w  x_AFw  ta_af=5w  T_pR 
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VA-AFIT  Measures,  Preferential  Independence,  Weights/10  Feb  99 

AFRL/VA:  Dick  Colclough,  Mike  Ziegler,  Brian  Van  Vliet,  A1  Boudreau,  Yvette  Weber 
AFIT/ENS:  Capt  Winthrop 

Subject:  Warfare  Advantage 

Comments:  The  group  did  not  feel  they  had  the  expertise  to  accurately  estimate  the  value 
increment  changes  for  each  level  in  the  measure.  This  evaluation  measure  has  equal 
value  interval  levels.  Any  technology  that  can  be  linked  to  a  system  that  achieves  a 
certain  level  in  this  measure  will  have  the  same  level.  An  example  is  an  improved 
landing  gear.  By  itself,  the  landing  gear  has  no  value  to  the  warfighter.  On  the  right 
vehicle,  it  contributes  to  the  vehicles’  ability  to  neutralize,  deter,  overwhelm,  etc.  an 
adversary. 

Subject:  Mutual  Preferential  Independence 

Comments:  A  matrix  for  pairwise  comparing  each  evaluation  measure  against  all  of  the 
others  was  developed  to  establish  mutual  preferential  independence  was  developed.  Each 
row  compared  one  evaluation  measure  with  all  other  evaluation  measures  in  the  value 
model.  The  evaluation  measure  was  set  to  a  particular  level.  The  other  measures  being 
compared  with  were  set  to  an  arbitrary  low  level  and  an  arbitrary  high  level.  The  group 
was  asked  if  they  would  still  prefer  the  high  level  to  the  low  level  given  the  other 
evaluation  measure  is  fixed  at  a  particular  level.  They  were  asked  if  there  preferences  for 
the  levels  of  any  evaluation  measure  would  change  given  one  evaluation  measure  is  set  to 
a  various  fixed  values.  The  group  responded  they  would  not  change  their  preferences. 
This  test  validates  that  the  mutual  preferential  independence  assumption  holds  for  the 
value  model. 

Subject:  Weights 

Comments:  Below  the  weight  elicitation  process  is  summarized.  Those  weights  that  are 
struck  out  are  because  an  evaluation  measure  was  eliminated  from  the  value  model.  L/D 
and  Weight  Reduction  along  with  the  Mission  Effective  value  were  removed  because 
these  items  are  design  parameters.  They  are  design  parameters  while  the  other 
parameters  look  at  changes  in  operational  capabilities.  Standoff  was  eliminated  because 
it  is  covered  by  survival.  Standoff  was  originally  used  to  establish  the  relationship 
between  values  by  the  VA  head  office.  It  is  assumed  Survival  can  replace  Standoff  using 
the  same  relationship  originally  stated  for  Standoff.  The  Firepower  evaluation  measure  is 
believed  to  be  a  part  of  Effectiveness.  Mr.  Jim  Grove  (x5-3538)  plans  to  provide 
additional  information  that  may  change  the  definition  of  Effectiveness  slightly  to  cover 
the  concept  Firepower  would  have  covered. 


Reach/Gain 

1 

Regeneration 

Missions  Enabled 

2 

Regeneration 

= 

Response  Time 

Regeneration 

= 

Availability 

4 

= 

Availability 

Reach/Gain 

170 


§ 

ss 

.  Ms 

Reach/Sustain 

Wi 

|  |p!  . .  ill  |i|  !J|i  Up  i|||  ft  " 

6 

1.5  *  Missions  Enabled 

Achieve/Extend  Life 

7 

1.2*  Missions  Enabled 

= 

Missions  Enabled 

8 

1.5  *  Missions  Enabled 

= 

Availability 

9 

1 .5  *  Missions  Enabled 

= 

Regeneration 

10 

1 .7  *  Missions  Enabled 

= 

Safety 

"  '  ' 

Awareness/Gain 

y 

. (1 

11 

Regeneration 

= 

Availability 

12 

Regeneration 

= 

Missions  Enabled 

Power/Lethality 

■ 

|  Hi  | 

1 1  71  \  f  0 1 

U  Ul  T  X  T  Clf 

Fircpov.-cr 

14 

1.3  *  Survival 

= 

Effectiveness 

4# 

V.  l  ~|  r  -j  -^  r  q  ] 

0W1  V  I!  ▼  Ctl 

= 

LU-HUV/tl- 

IllliS 

Tech.  Sup./Be  Efficient 

pj 

i '  ,  ■  ■  r  .1:  ■■  j  i  i  ;  ;  'r~ 

16 

1 .4  *  Dual  Use 

= 

Intermediate  Products 

17 

1.8  *  Dual  Use 

= 

Uniqueness 

18 

1 .5  *  Dual  Use 

= 

Multiuse 

19 

2  *  Dual  Use 

= 

User  Rating 

« 

Tech.  Sup./Expand  Our 

Frontier 

■ 

20 

Revolutionary 

= 

New  Warfighting  Spectrum 

Tech.  Sup./Simplify  Ops. 

* 

21 

Ratio  of  People 

= 

Interoperability 

22 

Ratio  of  People 

= 

Enables 

Tech.  Sup./Win 

23 

1.2  *  Cost  to  Adversary 

= 

Lead 

24 

1.5  *  Cost  to  Adversary 

= 

Warfare  Advantage 

Subject:  Weights  for  Reach/Gain,  Awareness/Gain,  and  Tech.  Superiority/Simplify 
Operations 

Comments.  The  Integrated  Concept  Leaders  found  themselves  splitting  up  on  which 
evaluation  measures  were  more  important.  Under  Reach/Gain,  the  Space  ICL  thought  the 
evaluation  measures  ranked  by  importance  was:  1)  Missions  Enabled,  2)  Regeneration  3) 
Response  Time,  and  4)  Availability.  The  UAV  ICL  felt  the  order  was:  1)  Availability,’ 2) 
Response  Time,  3)  Missions  Enabled,  and  4)  Regeneration  (which  agrees  with  the  Air 
Vehicles  head  office,  incidentally). 

Subject:  L/D  and  Weight  Reduction  under  Reach 

Comments:  Removed  from  model  because  it  did  not  “fit”  with  the  other  measurements. 
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VA-AFIT  Concerns  over  some  Measures  and  Weights/1 9  Feb  99 

AFRL/VA:  Dick  Colclough,  Mike  Ziegler,  A1  Boudreau 
AFIT/ENS:  Capt  Winthrop,  LTC  Kloeber,  Dr.  Deckro 

Subject:  L/D  and  Weight  Reduction 

Comments:  L/D  and  Weight  Reduction  do  not  fit  in  the  model,  but  something  is  missing 
if  it  is  removed.  The  group  agreed  to  add  a  combined  measure  called  “Payload-Range” 
which  would  measure  the  effects  of  both.  This  can  be  done  because  payload  and  range 
directly  trade  off  with  each  other.  For  more  details  on  this,  see  Appendix  E:  Value 
Hierarchies  and  Means  Networks. 

Subject:  Remove  Weight  Reduction  from  Sustain? 

Comments:  The  group  decided  to  leave  this  evaluation  measure  in  the  hierarchy. 
Sustainment  actions  can  result  in  changes  in  weight  for  existing  vehicles,  but  rarely  is  it 
enough  to  significantly  impact  the  payload  or  range  of  the  vehicle. 

Subject:  Survival 

Comments:  The  question  of  whether  or  not  a  separate  value  for  human  survival  and 
weapon  system  survival  was  brought  up.  A  person  flying  an  Air  Vehicle  is  at  far  less  risk 
than  a  person  in  a  manned  aircraft  during  combat.  Air  Vehicles  felt  survival  should 
encompass  both  the  human  element  and  the  weapon  system  element.  Hence,  this 
measure  was  not  changed. 

Subject:  Weights  for  evaluation  measures  under  Reach  under  Gain 

Comments:  The  weights  for  all  of  the  measures  under  were  determined  to  be  equally 

weighted. 
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VA-AFIT  Scoring/22  Feb  99, 24  Feb  99  -  25  Feb  99. 

AFRL/VA:  Doug  Dolvin,  Jim  McDowell,  Forrest  Sandow 
AFIT/ENS:  Capt  Winthrop,  LTC  Kloeber 

Comments:  The  22  Feb  99  meeting  established  the  scoring  team  that  would  use  the  Air 
Vehicles’  model  for  the  first  time  on  real  programs. 

Subject:  Scoring  Results  and  Notes 
Begin  with  measures  under  sustain 
Forrest  Sandow  -  Doug  Dolvin 

Weapons  Bay  Door  Noise  Suppression  -  Alternative  A 

Composite  Repair  of  Aircraft  Structures  -  Alternative  B 

IT AC  -  Alternative  C 


Number  of  Missions  enabled  (sustain  under  reach) 


A 

B 

C 

D 

E 

F 

2 

0 

0 

Comments:  A-  Active  suppression  allows  larger  Mach  number,  which  enables  a  lot 

more  missions 


B:  Just  fixing  structures  so  no  new  missions 
Other  comments:  Maybe  new  missions  shouldn’t  be  included  in  sustain  column. 


Achieve  Extend  Life  (sustain  under  reach) 
Years  until  the  system  must  be  replaced 


A 

B 

C 

D 

E 

F 

TO=Tn 

1.5 

2 

Comments:  A-no  change  in  design  but  possibly  slight  increase  in  life  due  to  current 

out-of-range  use 


B-  Mainly  large  transport  A/C,  future  in  fighters 
E-  Doubling  useable  life  by  monitoring  condition.  No  routine 
maintenance. 


Availability  (sustain  under  reach) 


A 

B 

C 

D 

E 

F 

1 

42  vs  3  over 

Eszsssns 
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life 

from  Gain 

Comments:  A-just  about  a  push  -  more  electronics  -  less  mechanical  systems  -  assume 

stay  within  boundaries  of  speed 


B-  bonded  repairs  are  more  permanent  so  time  to  repair  is  longer  short 

term 


fatigue  cracks,  corrosion,  different  aircraft,  different  systems-large 

transport 

with  ->  3  days  and  good  for  life  of  aircraft 

without  ->  1  day  and  good  for  5  years  then  6  weeks  for  part  replacement 
Looks  like  42  days  without  and  3  days  with. 

E-  No  change  because  it  is  an  add  on  technology,  same  effect  either  way. 


Regeneration 


'ime  (sustain  under  reach) 


A 

B 

C 

D 

E 

F 

1 

1 

50%  decrease 

Comments:  A-No  effect 

B-No  effect,  especially  with  combat  damage-  long  term  fatigue 
E-See  Gain 


Decrease  in  #  of  class  A  incidents  per  flight  hour(sustain  under  reach) 


A 

B 

C 

D 

E 

F 

No  effect 

No  effect 

No  Change 

Comments:  A- 

B- 


Comments  other:  Decrease  in  probability  of  occurrence  -10-7  to  10-7 


Weight  Reduction  (sustain  under  reach) 


A 

B 

C 

D 

E 

F 

No  effect 

No  effect 

Negligible 

Comments:  A-too  little  to  count 


B-too  little  to  count 
Comments  other: 


Only  A-Weapons  Bay,  C-ITAC  ,D-SITE-M,E-HMS  ,F-SB  System,  G-MTV 
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Availability  (gain  under  reach) 


A 

B 

C 

D 

E 

F 

G 

No  effect 

20%  decrease 

10%  increase 

7  times?? 

15% 

Pn=80% 

Comments:  A-no  effect  in  domain 

C-  system  compared  -  6~>  F-16D  same  aircraft  being  used 
D-  system  compared  -  UCAV  Lethal  C 

E-system  compared  -  F-15E/F-16  -  assume  65%  now-  71.5%  with 
F-  system  compared  -  Space  shuttle  -  2/13  -Harry  Karosopoulos  - 


flight/per  week 


G-system  compared  -  F-18EF/F-22-  old  is  65%  without  hydraulic 


tech 
8  hours 

systems 


Comments  other:  P  stands  for  percent  ready. 

Missions  enabled(gain  under  reach) 

Only  A-Weapons  Bay,C-ITAC  ,D-SITE-M,E-HMS  ,F-SB  System,G-MTV 


A 

B 

C 

D 

E 

F 

G 

2 

2 

0 

2 

0 

Comments:  A-no  effect  in  domain  -  high  speed  missions  not  currently  available 

C-  system  compared  -  6->  F-16D  same  aircraft  being  used 
D-  system  compared  -  UCAV  Lethal  C  -  covert  strike  -  main  benefit 
E-system  compared  -  F-15E/F-16  -  long  endurance  -  Chernobyl  -  SEAD 
F-  system  compared  -  Space  shuttle  - 
G-system  compared  -  F-18EF/F-22- 


Comments  other  -  more  than  two  new  missions  for  both  high  scorers. 


Only  A,C,D,E,F,G 
Regeneration(gain  under  reach) 

Only  A-Weapons  Bay,C-ITAC  ,D-SITE-M,E-HMS  ,F-SB  SystemSMV,G-MTV 


A 

B 

C 

D 

E 

F 

G 

no  effect 

no  effect 

no  effect 

50%  decrease 

50%  decrease 

25%  decrease 

Tn=18  Hours 

Comments:  A-no  effect  in  domain 


C-  system  compared  -  6->  F-16D  same  aircraft  being  used 
D-  system  compared  -  FI  17A-  extra  20  minutes  but  concurrent 
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E-system  compared  -  F-l  6  -  central  hydraulic  system  decrease 
regeneration  time  due  to  damage  assessment.  Eliminates  Flight  Control  System  can  not 
duplicates. 

F-  system  compared  -  Space  shuttle  -  2  days  for  new  vs  30  days-  Harry 
Karosopoulos-2nd  floor  middle  aisle-243 

G-system  compared  -  F-18EF/F-22-  combat  repair  time  is  much  better  - 
lost  hydraulic  system  and  modular  LRUs-2  level  maintenance  scheme 


Comments  otherOnly  A,C,D,E,F,G 


Response  Time  (gain  under  reach) 

Only  A-Weapons  Bay,C-ITAC  ,D-SITE-M,E-HMS  ,F-SB  System, G-MTV 


A 

B 

C 

D 

E 

F 

G 

No  effect 

■ 

no 

increase 

20%  less 
time 

No 

improvement 

No 

Improvement 

20%  less 
time 

tn  =  90  Minutes 

Comments:  A-no  effect  in  domain 

C-  system  compared  -  6-^  F-16D  same  aircraft  being  used->48  hours  to 


48  hours  (800  mile  range)  assume  no  refueling  available  for  ITAC  prepositioned  people 
and  prepositioned  fueling 

D-  system  compared  -  FI  17A  -  in  theater  logistics  setup  -  otherwise  no 

effect. 

E-system  compared  -  F-15E/F-16  - 

F-  system  compared  -  Space  shuttle  -  Precision  covert  strike  but  weapons 

delivery  only 

G-system  compared  -  F-18EF/F-22—  in  theater  logistics  setup  -  otherwise 

no  effect 


Comments  other:  This  is  speed  based  only-assuming  CONUS  base.  Assuming  no 
logistics  tail. 


Payload-Range  (gain  under  reach) 

Only  A-Weapons  Bay,C-ITAC  ,D-SITE-M,E-HMS  ,F-SB  System,  G-MTV 


A 

B 

C 

D 

E 

F 

G 

No  effect 

20% 

No  improvement 

10% 

10%  increase 

Comments:  A-no  effect  in  domain 

C-  system  compared  -  6-^  F-16D  same  aircraft  being  used-> trajectory 
optimization  ITAC(geese  flying) 
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D-  system  compared  -  FI  17A  -  in  theater  logistics  setup  -  otherwise  no 

effect. 

E-system  compared  -  F-15E/F-16  - 

F-  system  compared  -  Space  Operating  vehicle  (replace  all  other 
measures)  - 10%.  6000  lbs  to  LEO,  1000  lbs  to  Polar  orbit,  25,000  lbs  “popup”  to  LEO 
G-system  compared  -  F-18EF/F-22-  F22-  10%-;weight  reduction 


Comments  other:  This  is  speed  based  only-assuming  CONUS  base.  Assuming  no 
logistics  tail. 


Availability  (Awareness/Gain) 

Only  A-Weapons  Bay,  C-ITAC  ,D-SITE-M,E-HMS  ,F-SB  System,  G-MTV 


A 

B 

C 

D 

E 

F 

G 

No 

Change 

No 

Change 

No 

Change 

No 

Change 

No 

Change 

No 

Change 

Comments: 


Comments  other: 

Missions  Enabled(Awareness/Gain) 

Only  A-Weapons  Bay,  C-ITAC  ,D-SITE-M,E-HMS  ,F-SB  System,  G-MTV 


A 

B 

C 

D 

E 

F 

G 

No 

Change 

No 

Change 

■ 

No 

Change 

No 

Change 

No 

Change 

Comments: 


Comments  other: 

Regeneration  (Awareness/Gain) 

Only  A-Weapons  Bay, C-ITAC  ,D-SITE-M,E-HMS  ,F-SB  System,G-MTV 


A 

B 

C 

D 

E 

F 

G 

No 

Change 

No 

Change 

No 

Change 

Yes 

No 

Change 

No 

Change 

Comments: 


Comments  other: 

Effectiveness  (Power/Lethality) 

Only  A-Weapons  Bay, C-ITAC  ,D-SITE-M,E-HMS  ,F-SB  System,G-MTV 
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A 

B 

C 

D 

E 

F 

G 

No  Change 

No  Change 

20%  Improvement 

0 

Comments: 


A- 

C  -  This  value  is  a  guess.  Can  send  in  4  UAV’s  and  they  can  get  closer  for  better 
accuracy  with  less  risk  to  a  person. 

D  -  No  Change 

Comments  other: 

Survival  (Power/Lethality) 

Only  A-Weapons  Bay,C-ITAC  ,D-SITE-M,E-HMS  ,F-SB  System,G-MTV 


A 

B 

C 

D 

E 

F 

G 

10% 

Improvement 

No 

Change 

20% 

Improvement 

50% 

Improvement 

N 

0 

N 

0 

%15 

decrease 

Comments: 


A  -  Uncertain  of  actual  effect. 

C  -  This  value  is  a  guess. 

D  -  Low  Observable  compared  to  F-l  17  due  to  better  technology. 

G  -  Based  on  analysis  of  aircraft  returning  with  combat  damage,  identification  of  where 
damage  was  not  taken  estimates  the  vulnerable  areas  of  the  aircraft  (i.e.  those  aircraft  did 
not  return).  Elimination  of  hydraulics  eliminates  a  single  point  vulnerability  that  brings 
down  aircraft. 

Comments  other: 


AF  Savings  Ratio  (Tech.  Superiority) 

Only  A-Weapons  Bay,C-ITAC  ,D-SITE-M,E-HMS  ,F-SB  System,G-MTV 


A 

B 

C 

D 

E 

F 

G 

4/1 

3/1 

10/1 

0 

S35M/20  Years 

$25B/20  Years 

$1 0B/20  Years 

$7M 

$6M 

$11M 

$2M 

$4M 

$25  M 

$25  M 

Comments: 

A-  Design  of  spoilers  is  challenging.  Initial  design  is  cheaper.  This  number  is  a  best  case 
for  Gain.  Retrofitting  is  a  lot  harder. 

C-$l  1M  from  French  and  $1 1M  from  Air  Vehicles.  Slightly  less  than  $10M  ($9.8M) 
from  Air  Vehicles.  Requires  an  ATD  before  this  can  actually  be  used.  Replacing  a 
squadron  of  F-16’s  with  unmanned  F-16’s 

D  -  Savings  for  this  is  negative.  12%  more  than  a  baseline  system. 
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Comments  other: 


Dual  Use  (Tech.  Superiority/Be  Efficient) 

Only  A-Weapons  Bay,C-ITAC  ,D-SITE-M,E-HMS  ,F-SB  System,G-MTV 


A 

B 

C 

D 

E 

F 

G 

NO 

Yes 

Yes 

No 

Yes 

No 

Yes 

Comments: 

B  -  Airlines  interested  along  with  FAA. 

C  -  Border  Patrol  people  interested  in  the  program. 


Comments  other: 

Intermediate  Products  (Tech.  Superiority/Be  Efficient) 

Only  A-Weapons  Bay,C-ITAC  ,D-SITE-M,E-HMS  ,F-SB  System, G-MTV 


A 

B 

C 

D 

E 

F 

G 

No 

Yes 

Yes 

No 

Yes 

Comments: 

C  -  Questionable  about  12  months.  Might  be  24  months. 

Comments  other: 

Multiuse  (Tech.  Superiority/Be  Efficient) 

Only  A-Weapons  Bay,C-ITAC  ,D-SITE-M,E-HMS  ,F-SB  System,G-MTV 


A 

B 

C 

D 

E 

F 

G 

2  IC 

1  IC 

2  IC 

1IC 

3IC 

1IC 

2IC 

Comments: 

A  -  UAV,Sustainment 
C  -  UAV,  Space 


Comments  other: 

Uniqueness  (Tech.  Superiority/Be  Efficient) 

Only  A-Weapons  Bay,C-ITAC  ,D-SITE-M,E-HMS  ,F-SB  System,G-MTV 


A 

B 

C 

D 

E 

F 

G 

Partner  (Austrailia) 

Partner  (Cost 
Share  w/  FAA) 

Partner 

Partner 

Part 

ner 

Part 

ner 

Partn 

er 

Comments: 
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Comments  other: 


User  Rating  (Tech.  Superiority/Be  Efficient) 

Only  A-Weapons  Bay,C-ITAC  ,D-SITE-M,E-HMS  ,F-SB  System, G-MTV 


A 

B 

C 

D 

E 

F 

G 

No  Rating 

Endorsed 

No 

Rating 

No 

Rating 

JSF 

Endorsed 

ATD 

NASA 

endorsed 

Comments: 


B  -  Not  top  10  ATD 
Comments  other: 

Revolutionary  (Tech.  Superiority/Expand  our  Frontier) 

Only  A-Weapons  Bay,C-ITAC  ,D-SITE-M,E-HMS  ,F-SB  System,G-MTV 


A 

B 

C 

D 

E 

F 

G 

No 

No 

No 

No 

Yes 

Yes 

Yes 

Comments: 

C  -  Yes  to  warfighter. 


Comments  other:  Should  be  redefined  to  be  technology  that  changes  the  Warfighter’s 
operation. 

New  Warfighting  Spectrum  (Tech.  Superiority/Expand  our  Frontier) 

Only  A-Weapons  Bay,C-ITAC  ,D-SITE-M,E-HMS  ,F-SB  System, G-MTV 


A 

B 

C 

D 

E 

F 

G 

No 

Yes 

No 

No 

No 

Comments: 

C-  Cooperative  Ops.  Flexibility  is  greater.  Training  of  pesonell  is  less.  Allows  quick 
changes  in  posture. 

D-  Covert  ingress  allows  surpise. 

Comments  other: 

Programmatic  Risk(Tech.  Superiority) 

Only  A-Weapons  Bay,C-ITAC  ,D-SITE-M,E-HMS  ,F-SB  System,G-MTV 


A 

B 

C 

D 

E 

F 

G 

Low  (90-10) 

Low  (80-20) 

Low  (80-20) 

Low  (75-25) 

Low 

High 

Med 
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Comments:  If  the  money  is  available,  there  should  be  no  problem.  Risk  is  whether  or  not 
the  money  will  be  available. 

A,  B  Tech.  Risk  Low. 

C  -  20%  chance  of  not  completing  all  of  the  goals.  Medium  Tech.  Risk. 

D  -  25%  chance  of  not  completing  all  of  the  goals.  Medium  Tech.  Risk. 

Comments  other:  Tech,  risks  should  be  a  separate  factor. 

Enables  (Tech.  Superiority/Simplify  Operations) 

Only  A-Weapons  Bay,C-ITAC  ,D-SITE-M,E-HMS  ,F-SB  System,G-MTV 


A 

B 

C 

D 

E 

F 

G 

Yes 

No 

Yes 

Yes 

Yes 

Yes 

Yes 

Comments: 

Assume  operations  does  not  include  Maintenance. 

B  -  Change  to  Yes  if  Maintenance  is  included. 

C-  Co-operative  effort,  collision  avoidance. 

D  -  Covert  Ingress  is  enabled. 

Comments  other: 

Interoperability  (Tech.  Superiority/Simplify  Operations) 

Only  A-Weapons  Bay,C-TTAC  ,D-SITE-M,E-HMS  ,F-SB  System,G-MTV 


A 

B 

C 

D 

E 

F 

G 

Coalition 

All  Services 

Coalition 

All  Services 

Coal 

USAF 

Coal 

Comments: 

A-  Australian  Connection 
C  -  French  Connection 


D  -  USAF,  Navy,  JTCG 
Comments  other: 

Ratio  of  People(Tech.  Superiority/Simplify  Operations) 

Only  A-Weapons  Bay,C-ITAC  ,D-SITE-M,E-HMS  ,F-SB  System, G-MTY 


A 

B 

c 

D 

E 

F 

G 

1'^  fflUi 

1/3 

1 

1/3 

2/3 

Comments: 


B-  Difficult  to  ratio  people  reduction.  Does  not  include  changes  in  Depot  Maintenance. 
C-Losing  4  pilots.  Can  be  better  with  larger  numbers  of  unmanned  vehicles. 
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D-Adds  people.  Look  at  Maintenance  crew  for  117  versus  Maintenance  crew  +2 
Comments  other: 

Cost  to  Adversary  (Tech.  Superiority/Win) 

Only  A-Weapons  Bay,C-ITAC  ,D-SITE-M,E-HMS  ,F-SB  System,G-MTV 


A 

B 

C 

D 

E 

F 

G 

Medium 

Low 

Low 

■srasw 

Comments: 


C-  Developing  for  UCAV  would  be  very  expensive. 
Comments  other: 


Lead  (Tech.  Superiority/Win) 

Only  A-Weapons  Bay,C-ITAC  ,D-SITE-M,E-HMS  ,F-SB  System,G-MTV 


A 

B 

C 

D 

E 

F 

G 

5-10  Years 

<  1  Year 

5-10  Years 

5-10  Years 

1-5  Years 

Comments: 


A  -  Counteracting  is  not  known. 

B  -  Tech,  being  Given  Away 

c 

D- 

Comments  other:  Intervals  may  be  too  large  (1-2  years) 


Warfare  Advantage  (Tech.  Superiority /Win) 

Only  A-Weapons  Bay,C-ITAC  ,D-SITE-M,E-HMS  ,F-SB  System, G-MTV 


A 

B 

C 

D 

E 

F 

G 

Degrade 

No  Effect 

Neutralize 

Keep  Pace 

| 

Comments: 


Comments  other: 
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Appendix  C:  Evaluation  Measure  Templates  and  Macros 
Evaluation  Measure  Types 

A  set  of  Excel  templates  and  macros  were  developed  to  simplify  the  development 
of  the  evaluation  measures.  Evaluation  measures  types  implemented  were  the  discrete 
measure,  exponential  measure,  piecewise  fit,  and  S-curve.  Each  type  is  discussed,  in 
turn. 


— 

Uniqueness 

iTITTnTSfli 

Value 

Category  Definition 

0 

No  Effect 

0.00 

Technology  that  is  ineffective  against  an  adversary. 

1 

Technology  that  is  needed  to  stay  at  the  same  level  as  an  adversary. 

2 

■BSES3 

Technology  that  results  in  a  tactical  advantage  for  our  forces. 

3 

Neutralize 

60.00 

Technology  advances  that  render  some  or  all  of  an  adversaries  military  capability  ineffective  and 
cannot  be  countered  at  the  completion  of  the  advance 

4 

Deter 

Technology  that  discourages  an  adversary  from  actinq  because  of  doubt,  fear  or  cost. 

5 

100 

Technology  that  makes  it  impossible  for  an  adversary  to  act. 

Figure  24:  Sample  Discrete  Evaluation  Measure  Calculation  Sheet 

Figure  24  shows  a  sample  discrete  measure.  The  discrete  measure  is  used  when 
several  categories  exist  that  describe  the  attribute.  The  calculation  sheet  has  the  ability  to 
graphically  display  each  category  and  its  value.  Each  category  has  a  name  defined  below 
the  graphical  display,  has  a  value  associated  with  it,  and  a  has  a  detailed  definition 


associated  with  the  name.  The  “comments”  block  can  be  used  to  record  information  from 


the  expert  providing  the  information. 

The  Excel  sheet  has  the  capability  of  automatically  rescaling  the  value  axis  or  y- 
axis  to  the  value  scale  selected  by  the  user.  This  is  accomplished  using  an  Excel  macro, 
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which  is  activated  by  the  “ReScale”  button.  Rescaling  the  X-axis  is  accomplished  by 
adding  or  subtracting  evaluation  measure  categories  and  then  changing  the  data  series 
references  of  the  graph.  The  “copy  sheet”  button  for  all  evaluation  measures  will  be 
discussed  later. 


Evaluation  Measure: 

Definition: 

Comments: 


Value  Function  Range  | 

115.7012327  115.7012327  30.041  0,00  0.0 

Lower 

. o 

Set  Value 

50 

100 

Midpoint  for  straight  Line 


Guess  for  R 


Graph  Interval 


Copy  Sheet 


Figure  25:  Sample  Exponential  Evaluation  Measure  Calculation  Sheet 

Figure  25  shows  the  Excel  sheet  to  use  for  exponential  functions.  The  values 
under  the  X  column  and  the  values  under  “Value  Function  Range”  for  “Lower”  and 
“Upper”  can  be  changed  by  the  user.  Setting  values  in  the  X  column  are  used  to 
determine  the  lower,  upper  and  midvalue  area  where  the  change  in  value  from  the  lower 


to  the  midvalue  point  is  the  same  change  in  value  as  from  the  midvalue  point  to  the  upper 
value  level.  Pressing  the  “Find  R”  macro  button  automatically  calculates  the  values  for 
A,  B  and  R  using  numerical  methods  supported  by  Excel.  The  “Lower”  and  “Upper” 
values  under  “Value  Function  Range”  can  be  changed  to  any  range.  This  allows 
rescaling  of  the  value  axis  or  y-axis,  which  takes  place  automatically  when  “Find  R”  is 
pressed.  Additionally,  it  allows  the  development  of  curves  over  smaller  ranges.  This 


184 


capability  allows  the  analyst  to  fit  several  exponential  curves  together  to  develop  a 
“piecewise  exponential”  curve. 

The  equations  used  in  deriving  value  for  the  exponential  function  were: 

v(jc)  =  A- B* exp(-xM  / R ) 

A  =  L+B*exp(-xL/R)  Equation  10 

B. - Mzk - 

exp(-*t  /  R)  -  exp(-xw  /  R) 

where  L  is  the  lowest  value  to  be  considered  for  the  function  and  H  is  the  highest  value 
being  considered  for  the  function.  The  variable  x  is  a  possible  level  in  the  measure  where 
xl  is  the  lowest  level  of  the  measure  possible,  xh  is  the  highest  level  of  the  measure,  and 
xm  is  the  value  where  the  decision  maker  equally  prefers  a  change  in  value  from  the 
increment  xl  to  xm  as  the  increment  from  xm  to  xh-  The  value  R  is  calculated  using 
Excel’s  “Goal  Seek”  command.  The  variable  v(x)  is  set  to  the  value  being  searched  for 
by  the  numerical  method. 

Figure  26  shows  a  sample  Excel  sheet  for  calculating  piecewise  linear  functions. 
The  Excel  graph  automatically  displays  each  function.  The  analyst  simply  inputs  values 
in  the  X  and  Y  column  and  the  slope  and  intercepts  are  automatically  calculated.  The 
analyst  can  input  scale  values  for  “Minimum”  and  “Maximum”  in  the  X  and  Y  column. 
By  hitting  the  “ReScale”  button  in  the  sheet,  a  macro  is  invoked  which  automatically 
rescales  the  graph  based  on  the  scaling  values  that  were  entered  into  the  Excel  sheet.  A 
section  is  provided  in  the  Excel  sheet  to  identify  the  evaluation  measure,  define  it,  and 
provide  comments  during  the  elicitation  process. 
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Evaluation  Measure: 

Definition: 

Comments: 


Figure  26:  Sample  Piecewise  Linear  Calculation  Sheet 


Figure  27:  Sample  S-Curve  Calculation  Sheet 

Figure  27  shows  a  sample  sheet  for  calculating  an  S-curve  evaluation  measure. 
The  S-curve  allows  the  analyst  to  enter  the  lowest  value  and  highest  value  for  the  curve 
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under  the  “X”  column.  Slider  bars  allow  the  analyst  to  change  the  bias  and  the  steepness 
of  the  curve.  Changing  the  “Graph  Y-Axis  Scale  Constant”  can  rescale  the  graph  value 
and  hitting  the  “Rescale”  button,  automatically.  Additionally,  it  is  possible  to  calculate 
“T”  from  elicited  points  with  a  decision-maker.  All  the  analyst  does  is  to  type  in  the 
elicited  points  in  the  elicited  point  column  and  the  value  of  T  is  automatically  calculated. 

The  “Copy  Sheet”  button  copies  the  current  sheet  to  a  new  Excel  Workbook 
called  “VA  Value  Model.xls.”  The  Excel  sheets  shown  above  were  developed  an  placed 
in  an  Excel  workbook  called  “Measure  Calc.xls.”  The  “Copy  Sheet”  button  allows  the 
analyst  to  develop  an  evaluation  measure  in  the  “Measure  Calc.xls”  workbook  and  then 
copy  the  evaluation  measure  over  to  the  “VA  Value  Model.xls”  workbook,  once  it  is 
completed.  This  allows  repeated  use  of  the  “Measure  Calc.xls”  tools  and  quick  saving  of 
the  results  in  the  new  workbook. 

Excel  Button  Macros 

Several  Excel  macros,  connected  to  buttons  in  Excel  sheets,  were  developed  to 
simplify  evaluation  measure  development.  What  these  macros  do  was  discussed  in  the 
last  section.  These  macros  were  developed  using  the  “Record  Macro”  option  in  Excel. 
They  were  modified  slightly,  in  some  cases.  Actual  macrocode  written  in  visual  basic 
and  a  short  explanation  of  their  function  is  shown  below: 

Sub  Find_R() 

t 

'  Find_R  Macro 

'  Macro  recorded  1 1/4/98  by  Michael  Winthrop 
'  Implemented  by  the  "Find  R"  button  in  the  Exponential  Excel  Sheet 

t 

Range("C23"). Select 
Selection.Copy 
Range("C18"). Select 
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Selection.PasteSpecial  Paste:=xlValues,  Operation:=xlNone,  SkipBlanks:=  _ 
False,  Transpose:=False 
Application.CutCopyMode  =  False 

Range("E19").GoalSeek  Goal:=Range("G19"),  ChangingCell:=Range("C18") 

ActiveSheet.ChartObjects("Chart  2").Activate 
ActiveChart.Axes(xl  Value).  Select 
With  ActiveChart.Axes(xlValue) 

.MinimumScale  =  Range("G18") 

.MaximumScale  =  Range("G20") 

.MinorUnitlsAuto  =  True 
.MajorUnit  =  Range("G22") 

.Crosses  =  xlAutomatic 
.ReversePlotOrder  =  False 
.ScaleType  =  xlLinear 
End  With 

Range( "  C 1 8 " ) .  S  elect 

End  Sub 

Sub  S_Curve_Rescale() 

i 

'ReScales  the  Y  axis  in  the  S-Curve  Excel  Sheet,  only. 

'The  Rescale  button  implements  this  macro. 

ActiveSheet.ChartObjects("Chart  12").  Activate 
ActiveChart.Axes(xlValue). Select 
With  ActiveChart.Axes(xl Value) 

.MinimumScale  =  0 
.MaximumScale  =  Range("D23") 

.MinorUnitlsAuto  =  True 
.MajorUnit  =  Range("G23") 

.Crosses  =  xlAutomatic 
.ReversePlotOrder  =  False 
.ScaleType  =  xlLinear 
End  With 

Range("C18"). Select 

End  Sub 

Sub  PL_Rescale() 

i 

'This  Macro  rescales  the  piecewise  linear  Excel  Graph  and  is  implemented  using 

'the  rescale  button  on  that  sheet. 

ActiveSheet.ChartObjects("Chart  1 ").  Activate 
Acti  veChart .  Axes(xl  V  al  ue)  .Select 
With  ActiveChart.Axes(xlValue) 
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.MinimumScale  =  Range("D24") 

.MaximumScale  =  Range("D25") 

.MinorUnitlsAuto  =  True 
.MajorUnitlsAuto  =  True 
.Crosses  =  xlAutomatic 
.ReversePlotOrder  =  False 
.ScaleType  =  xlLinear 
End  With 

ActiveChart.Axes(xlCategory).Select 
With  ActiveChart.Axes(xlCategory) 

.MinimumScale  =  Range("C24") 

.MaximumScale  =  Range("C25") 

.MinorUnitlsAuto  =  True 
.MajorUnitlsAuto  =  True 
.Crosses  =  xlAutomatic 
.ReversePlotOrder  =  False 
.ScaleType  =  xlLinear 
End  With 

Range("C18"). Select 

End  Sub 

Sub  D_Rescale() 

i 

This  macro  rescales  the  discrete  excel  sheet  automatically.  It  is  activated 

'by  hitting  the  rescale  button  on  the  sheet. 

ActiveSheet.ChartObjects("Chart  1 "). Activate 
ActiveChart.Axes(xlValue). Select 
With  ActiveChart.Axes(xlValue) 

.MinimumScale  =  0 
.MaximumScale  =  Range("G3”) 

.MinorUnitlsAuto  =  True 
.MajorUnitlsAuto  =  True 
.Crosses  =  xlAutomatic 
.ReversePlotOrder  =  False 
.ScaleType  =  xlLinear 
End  With 

End  Sub 

Sub  Copy_Sheet() 

t 

'This  macro  copies  the  current  sheet  to  an  new  workbook  called  "VA  Value  Model.xls" 

'By  changing  the  file  in  the  quotes,  other  files  can  be  copied  to. 

ActiveSheet.Copy  After:=Workbooks("VA  Value  Model.xls").Sheets(  _ 

1) 

End  Sub 


189 


Excel  Value  Function  Implementations 

Several  macros  were  developed  based  on  Kirkwood’s  implementation  of  value 

functions  (Kirkwood,  1997:  80-81).  Kirkwoods  “ValuePL”  function  was  used  as 

provided  in  his  book.  It  is  recopied  and  shown  below  for  reference.  Functions  were 

developed  to  calculate  an  exponential  curve,  an  S-Curve,  and  a  discrete  function. 

Additionally,  functions  were  developed  to  help  in  sensitivity  analysis,  allow  combining 

of  two  functions,  and  to  perform  interval  scale  transformations.  The  discrete  function 

curve  is  a  modified  version  of  Kirkwood’s  ValuePL  function.  The  exponential  function 

and  the  S-Curve  function  are  simple  implementation  of  the  equations  for  this  type  of 

function.  The  functions  in  visual  basic  code  are  shown  below: 

Function  Con_Ratio(weight_change,  weight_constant_ratio,  w_others) 

'  This  function  is  used  to  calculate  a  constant  weight  ratio  in  sensitivity 
'  analysis. 

Con_Ratio  =  (1  -  weight_change)  *  (weight_constant_ratio  /  w_others) 

End  Function 


Function  ValuePL(X,  Xi,  Vi) 

'  This  function  takes  piecewise  curves  and  interploates  between  the  points 
'  This  function  comes  from  Kirkwood 
i  =  2 

Do  While  X  >  Xi(i) 
i  =  i+  1 
Loop 

Z  =  Vi(i-l)_ 

+  (Vi(i)  -  Vi(i  -  1))  *  (X  -  Xi(i  -  1))  /  (Xi(i)  -  Xi(i  -  1)) 

If  (Z<0)  Then 
Z  =  0 
End  If 

If  (Z>  100)  Then 
Z  =  100 
End  If 

ValuePL  =  Z 
End  Function 

Function  ValueE2(X,  a,  B,  Rho) 

’Custom  function  for  calculating  the  value  of  an  exponential  function 
Z  =  a-  B  *  Exp(-X / Rho) 

If  Z  <  0  Then 
Z  =  0 
End  If 
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If  Z>  100  Then 
Z=100 
End  If 
ValueE2  =  Z 
End  Function 

Function  SCurve(X,  Value_Scale,  a,  B,  T) 

'Function  calculates  value  from  an  S-curve 
Z  =  (a  /  (a  +  Exp((-X  -  B)  /  T)))  *  Value_Scale 
If(Z<0)  Then 
Z  =  0 
End  If 

If  (Z  >  100)  Then 
Z=100 
End  If 
SCurve  =  Z 
End  Function 

Function  ValueDis(X,  Xi,  Vi) 

'Function  calculates  the  value  from  a  discrete  function 
i  =  1 

Result  =  "Mispelled"  '  Default  Output  of  function  if  no  match  is  made 
For  Each  Z  In  Xi 
If  X  =  Z  Then 

Result  =  Vi(i)  ’  A  match  was  made  -  determine  value 
Exit  For 
Else 

i  =  i  +  1  'No  match  was  made,  -  Check  the  next  one 
End  If 
Next 

ValueDis  =  Result 
End  Function 

Function  Tran(X,  Lower_Range,  Upper_Range,  Vu) 

'Function  transforms  value  over  a  range  to  be  between  0  and  an  upper  value,  Vu. 

i 

a  =  Vu  /  (Upper_Range  -  Lower_Range) 

B  =  -(Lower_Range  *  Vu)  /  (Upper_Range  -  Lower_Range) 

Tran  =  a  *  X  +  B 

End  Function 

Function  Two_F(Score,  XI,  X2,  X3,  Function_l,  Function_2) 

'Function  allows  use  of  two  functions  in  two  different  ranges 

i 

If  Score  >=  XI  And  Score  <  X2  Then 
Two_F  =  Function_l 
Else 

Two„F  =  Function_2 
End  If 

End  Function 
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Macros  for  integer  and  Linear  Programs  in  Excel 

Macros  were  developed  to  automatically  run  Solver  and  record  the  results.  For 
more  understanding  of  how  to  drive  Solver  using  Excel  Macros,  or  how  to  implement 
math  programs  with  solver,  see: 

Help  for  Microsoft  Excel  Solver  Users.  Frontline  Systems,  Inc.  On  Line. 
Available  at  http://www.frontsvs.com/xlhelp.htm. 

The  project  required  some  automation,  since  it  was  desired  to  optimize  a  portfolio  of 
programs  for  a  large  number  of  budgets.  The  macros  are  “hardwired”  to  run  with 
“Portfolio.xls.”  It  requires  the  decision  variables  (a  total  of  seven)  be  located  at  cell 
locations  “C16-C22”  on  the  excel  sheet.  It  requires  the  objective  function  to  maximize 
change  in  warfighter  support  due  to  technology  to  be  located  at  cell  “Cl 5”  and  requires 
the  constraint  “Investment  Cost”  be  located  at  cell  “C14.”  The  “Available  Budget”  is 
located  at  “C13.”  The  formula  in  “C13”  was  set  to  “D13-1.”  This  formula  propagates 
itself  as  a  side  benefit  of  the  way  the  macro  runs.  The  farthest  right  budget  cell  is  the 
highest  possible  budget  that  will  be  allocated. 

The  macro  runs  by  running  solver  using  the  variables  defined.  Solver  provides  an 
answer  by  changing  the  decision  variables  in  “C16-C22.”  The  macro  then  inserts  part  of 
a  column,  which  shifts  cells  “C13-C22”  right  one  column,  but  does  not  shift  any  other 
part  of  the  column.  The  macro  next  copies  “D13-D22”  into  the  new  cells  “C13-C22.” 

The  macro  is  then  complete.  By  restarting  the  macro,  the  user  can  calculate  the  policy  for 
the  new  conditions.  The  macro  for  solving  an  integer  program  is  shown  below: 

Sub  Run_Integer() 

'  Run_Solver  Macro 

'  Macro  recorded  3/7/99  by  Michael  Winthrop 

'  The  macro  has  been  modified  from  what  was  recorded  by  removing  redundant  lines. 

'  The  following  resets  the  solver,  adds  constraints,  sets  the  objective  function  and  runs  solver. 
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SoIverReset 

SolverOk  SetCell:="$C$15",  MaxMinVal:=l,  VaIueOf:="0H,  ByChange:="$C$16:$C$22" 
SolverAdd  CellRef:="$C$16:$C$22",  Relation:=5,  FormuIaText:="binary" 

SolverAdd  CellRef:=M$C$14M,  Relational,  FormulaText:="$C$13" 

SolverSolve  (True) 

'  The  following  moves  the  calculated  column  of  results  over  by  one  and  prepares 
’  the  new  column  for  the  next  calculation 
Range("C13:C22M). Select 
Selection. Insert  Shift:=xlToRight 
Range("D13:D22").Select 
Selection. Copy 
Range("C  1 3”). Select 
ActiveSheet.Paste 
Application.CutCopyMode  =  False 
Range("Cl  3"). Select 
End  Sub 

The  macro  for  solving  a  Linear  Program  is  shown  below: 

Sub  Run_LP() 

i 

'  Run_SoIver  Macro 

'  Macro  recorded  3/7/99  by  Michael  Winthrop 

i 

’  The  macro  has  been  modified  from  what  was  recorded  by  removing  redundant  lines. 

'  The  following  resets  the  solver,  adds  constraints,  sets  the  objective  function  and  runs  solver. 
SoIverReset 

SolverOk  SetCell:=”$C$15n,  MaxMinVal:=l,  ValueOf:=M0M,  ByChange:=,,$C$16:$C$22" 
SolverAdd  CellRef:="$C$16:$C$22",  Relational,  FormulaText:="=l" 

SolverAdd  CellRef:="$C$16:$C$22,\  Relation:=3,  FormulaText:="$E$2:$E$9" 

SolverAdd  CelIRef:="$C$14M,  Relational,  FormulaText:="$C$13" 

SolverAdd  CellRef:— '$C$16:$C$22",  Relation:=3,  FormulaText:=nFund_Program?" 
SolverSolve  (True) 

’  The  following  moves  the  calculated  column  of  results  over  by  one  and  prepares 
'  the  new  column  for  the  next  calculation 
Range("C  1 3  :C22n). Select 
Selection.Insert  Shift:=xlToRight 
Range("D13:D22"). Select 
Selection. Copy 
Range(MCl  3"). Select 
ActiveSheet.Paste 
Application.CutCopyMode  =  False 
Range(MC  1 3"). Select 
End  Sub 


It  is  possible  to  add  constraints  to  these  macros  or  change  the  constraints  by 
changing  or  adding  lines  that  say  “SolverAdd.”  The  “SolverOK”  command  sets  locations 
for  the  objective  function  information.  Use  of  the  “Record  Macro”  feature  of  Excel  (the 
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ways  these  macros  were  originally  developed)  can  have  problems.  A  problem  was  not  all 
of  the  constraints  were  added  into  solver  using  the  macro,  but  it  worked  fine  when  added 
manually.  The  problem  was  solved  when  it  was  realized  the  “Record  Macro”  routine  of 
Excel  had  made  a  syntax  error.  The  “FormulaText:=”  keyword  was  recorded  in  some 
constraints  was  recorded  as: 

FormulaText:=‘T” 
instead  of: 

FormulaText:=“=l” 

This  problem  had  to  be  fixed,  manually  for  the  macro  to  work. 
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Appendix  D:  S-Curves  as  a  Value  Measure 

The  S-curve  method  of  identifying  a  value  function  uses  the  following  equation: 


V(X)=- 


1 


Equation  11 


1+exp 


-X-B 


where  B  is  the  bias  of  the  curve  and  T  is  the  steepness  of  the  curve.  At  the  middle  value 
point  of  0.5  (assume  value  ranges  from  0  to  1),  the  value  of  B  =  -XM  where  XM  is  an 
arbitrary  measure  level  for  the  quantity  under  consideration.  Figure  28  shows  the  effects 
of  changing  the  bias  for  a  sample  S-curve  and  simple  algebra  proves  the  result: 


0.5  =  ■ 


1  +  exp 


-Xm~B 


0.5 

f 

1  +  exp 

r-x„-flY| 

T 

V 

^  1  )) 

=  1 


exp 


=  1  and  by  taking  the  natural  log  of  both  sides  and  simplifying 


b  =  -xm 

This  property  is  interesting  because  it  allows  determination  of  the  mid-value  from 
a  decision-maker  in  the  same  manner  as  elicitation  for  the  midvalue  of  the  exponential 
function.  A  second  point  could  be  elicited  between  the  midvalue  and  either  the  highest  or 
lowest  value  in  the  function  using  the  same  process.  Since  B  is  known,  it  would  be  a 
simple  matter  to  solve  for  T.  If  it  could  be  determined  from  the  decision-maker  that  there 
was  a  place  in  a  measure  where  value  increased  dramatically  and  further  changes  resulted 
in  little  improvement  in  value,  Equation  1 1  would  be  a  useful  one.  Figure  29  shows  the 
effects  of  changing  the  steepness  for  the  S-curve  for  a  generic  evaluation  measure. 
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Figure  28:  Examples  of  S-Curve  Single  Dimension 
Value  Functions  (Changes  in  Bias) 


Figure  29:  Examples  of  S-Curve  Single  Dimension 
Value  Functions  (Changes  in  Steepness) 
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Appendix  E:  Value  Hierarchies  and  Means  Networks 

An  essential  issue  in  developing  a  value  model  is  to  separate  values  from  the 
means  to  accomplish  the  value.  "A  means  objective  is  of  interest  in  the  decision  context 
because  of  its  implications  for  the  degree  to  which  another  (more  fundamental)  objective 
can  be  achieved.  Simply  stated,  the  means  objectives  are  important  because  they  are 
means  to  the  achievement  of  the  fundamental  objectives”  (Keeney,  1992:  34). 

The  relationship  among  adjacent  objectives  in  a  means  objective  network  is 
causal.  The  lower-tier  objective  is  a  means  to  the  higher-tier  objective.  The  means 
objectives  do  not  need  to  be  collectively  exhaustive  or  mutually  exclusive.  It  is  possible 
to  have  one  lower-tier  objective  below  a  higher  tier  objective.  Additionally,  means 
objectives  can  have  complex  interrelationships  (Keeney,  1992:  78). 

Factual  knowledge  is  used  to  construct  means  to  achieve  higher  tier  objectives. 
Value  judgements  are  used  to  construct  fundamental  objective  hierarchies  (Keeney,  1992: 
81).  When  disagreements  occur,  "it  is  easier  to  determine  whether  they  are  based  on 
different  facts,  different  judgements  about  facts,  different  values,  different  expressions  of 
value  judgements,  or  a  combination  of  these."  Separating  facts  and  values  makes  it 
possible  to  bring  those  in  who  are  knowledgeable  about  facts  and  those  knowledgeable 
about  values  separately.  Hence,  experts  in  facts  are  not  required  to  make  value 
judgements  and  experts  in  values  are  not  required  to  make  fact  judgements  (Keeney, 

1992:  93-94). 

The  fundamental  objectives  and  the  means  objectives  combined  allow  a  decision¬ 
maker  to  find  the  best  decisions.  According  to  Clemen,  the  means  networks  can  be  used 
in  three  ways.  First,  it  assures  the  decision-maker  that  fundamental  objectives  are  in  the 
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decision  model,  not  means  objectives.  Second,  an  easily  measured  means  objective  can 
sometimes  be  substituted  for  a  fundamental  objective  that  is  more  difficult  to  measure. 
Finally,  a  means-objective  network  can  be  used  to  generate  creative  new  alternatives 
(Clemen,  1996, 45). 

In  the  thesis,  a  means  network  was  not  specifically  developed.  Keeney  and 
Clemen  both  document  the  need  to  separate  values  and  means,  which  was  considered 
during  development  of  the  Air  Vehicles’  value  model.  For  example,  during  meetings 
with  Air  Vehicles,  a  few  of  the  evaluation  measures  identified  were  “not  at  the  same 
level”  as  the  others.  Evaluation  measures  originally  being  considered  under  “Reach” 
under  “Gain”  were  “Availability,”  “Missions  Enabled,”  “Regeneration,”  “Response 
Time,”  “Lift  to  Drag  Ratio,”  and  “Weight  Reduction.”  Air  Vehicles  did  not  believe  “Lift 
to  Drag  Ratio  and  “Weight  Reduction”  should  be  in  the  hierarchy,  but  something  that 
measured  changes  in  payload  and  range,  but  did  not  trade  off,  was  needed. 

The  reason  “Lift  to  Drag  Ratio”  and  “Weight  Reduction”  did  not  fit  under  Reach 
was  that  the  parameters  are  technical  measurements,  while  all  of  the  other  parameters  in 
the  hierarchy  are  performance  measurements.  Further,  if  the  “Why  is  this  important” 
question  is  asked  about  the  parameters,  the  answer  is  “because  they  are  the  means  to 
improving  payload  and  range  of  an  air  vehicle.”  While  Clemen  states  a  means  objectives 
can  be  used  as  a  proxy  measure  for  a  value,  Alexander  and  Mitchel  state  these  measures 
are  extremely  poor  for  measuring  performance.  Based  on  the  resistance  from  Air 
Vehicles  in  using  this  measure  combined  with  the  evidence  of  the  literature,  the  measures 
were  redefined  into  a  single  “Payload/Range”  constructed  measure. 
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Appendix  F:  Air  Vehicles’  Evaluation  Measures 


Figure  30:  Technological  Superiority/Air  Force  Savings  Ratio 
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Evaluation  Measure:  Simplify  Operations:  Enables _ 

Abreviation  I  Category  I  Value  _ Category  Definition 

XI _ No _ 0  Does  not  Simplify  Operations. _ 

X2  Yes _ 100  Simplifies  Operations. _ 


Figure  47:  Technological  Superiority/Simplify  Operations/Enables 
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Appendix  G:  Air  Vehicles’  Weight  Calculations 

Weights  for  Reach/Gain  evaluation  measures. 

Space  ranks  Reach/Gain  in  the  following  order:  1)  Missions  Enabled,  2)  Regeneration,  3) 
Response  Time,  4)  Availability.  UAV  ranks  Reach/Gain  in  following  order:  1) 
Availability,  2)  Response  Time,  3)  Missions  Enabled,  4)  Regeneration.  Sustainment 
ranks  same  as  UAV  except  switches  Regeneration  with  Missions  Enabled.  Payload- 
Range  was  added  later  and  was  not  ranked. 


wRG_Avail 

'.2* 

w  RG_Resp_Time 

.2 

w  RG_mis_enabled 

;= 

.2 

wRG_Regen 

.2 

w  RG_Pay_Rng 

.2 

Weights  for  Reach/Sustainment  evaluation  measures. 


wRS_Safety 

wRS_Ach_Ext 

A 

wRS_Avail 

.1 

w  RS_Regen 

.1 

w  RS_wt_red 

.1 

.1 

_w  RS_mis_enabled 

Given 

w  RS_Safety“  *-^‘w  RS_mis_enabled 
w  RS_Regen~  *  RS_mis_enabIed 
w  RS_AvaiF 1  *5w  RS_mis_enabled 
w  RS_AchJ3xf"  *  RS_mis„enabled 
w  RS_wt_red=  1  '2'w  RS_mis_enabled 
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w  RS_Safety  +  w  RS_Regen  +  w  RS_  Avail  +  w  RS_Ach_Exft" w  RS_wt_red  +  w  RS_mis_enabled=1 

w  RS_Safety 
w  RS_Ach_Ext 
wRS_Avail 

:-Find  (vv  rs  Safety  * w  RS_Ach_Ext’ w  RS  Avail’wRS  Regen’wRS  wt  red’wRS  mis  enabled) 
wRS_Regen  7 

w  RS_wt_red 

w  RS_mis_enabled  _ 

wRS  Safety  =0.202380952380952 
w  rs  wt  red  =0.142857142857143 

wRS_Ach_Ext=0-178571428571429 
wRS_mis_enabled=0-1 19047619047619 

wRS_Regen  =  0-178571428571429 
WRS  AvaiI  =0.178571428571429 

Weights  for  Awareness/Gain  evaluation  measures. 

'l 

w  RG_Avail  3 

1 

w  RG_mis_enabled  •“  ^ 

w  RG_Regen  _  i 

.3 

See  Reach/Gain  for  information. 

Weights  for  Power/Lethality 

wPL_eff:=-5 

wPL_survival :=-5 
Given 

w  PL_efF 1 3'w  PL_survival 
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w  PL_efT*" w  PL_survival“  ^ 
w  PL_eff 

:=FindwpL_eff,wpL_survival 

w  PL_survival  J 

WpL  eff=  0.565217391304348 

w  p^  survival  =  0-434782608695652 

Weights  for  Tech.  Sup.  Be  Efficient 

w  TBE_User 

wTBE_Multi  .1 

wTBE_Uniq  :=  1 

w  TBE_Int_Prod  1 

[.1 

wTBE_Dual_UseJ 

Given 

w  TBE_Int_Prod=  1  -4'w  TBE_Dual_Use 
w  TBE_Uniq=L8'w  TBE_Dual_Use 
w  TBE_Multf 1  -5w  TBE_Dual_Use 
w  TBE_User=2w  TBE_Dual_Use 

w  TBE_User+- w  TBE_MuIti+  w  TBE_Uniq+- w  TBE_Int_Prod  +  w  TBE_Dual_Use=  1 

wTBE_User 
w  TBE.Multi 

w  TBE_Uniq  := Find  (w  TBE_User’ w  TBE_Multi’ w  TBE_Uniq’ w  TBE_Int_Prod  ’ w  TBE_Dual_Use) 

w  TBE_Int_Prod 
_w  TBE_Dual_Use 

WTBE  User  =  0-25974025974026 

w  tbe  Int  Pi*ocl  ~  1818181818181 82 

w  TBE.Multi =  °- 1 94805194805 1 95 
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w  TBE_Dual_Use  =  0.12987012987013 

WTBE  Uniq  =  0-233766233766234 

Tech.  Superiority  Expand  our  Frontier  Weights 

wTEF_Rev 
wTEF_NWS 

Tech.  Superiority/Simplify  Operations 

'l 

wTSO_Enables  3 
1 

wTSO_ROP  ;=  “ 

w  TSO_Int  J  l 
3 

Tech.  Superiority/Win 
wTSW_War  :=1 
wTSW_Lead  :=A 
wTSW_CA  :~A 
Given 

w  TSW_Lead=  1  -2w  TSW_CA 
w  TSW_War=  1  Sw  TSW_CA 
w  TSW_Lead  +  w  TSW.CA  +  w  TSW_War=1 
wTSW_War 

w  TSW_Lead  :=  Find(w  TSW_War  > w  TSW_Lead  > w  TSW_Ca) 
wTSW_CA 

wTSw_War  =0.405405405405405 
w  TSW_Lead  =0.324324324324324 
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w  TSW  CA  =0.27027027027027 


Reach/Sustain/Achieve/Extend  Life  is  twice  as  important  as  Reach/Gain/ Availability. 
Sustainment  will  be  most  important  until  2002  when  gain  will  increase  in  importance. 
Purpose:  Relate  Reach/Gain  to  Reach/Sustain 

w  RS_Ach_Ext’w  R_Su’w  Reach=2'(w  RG_Avail’w  R_Gaw  Reach) 

Reduces  to: 

w  RS_Ach_Ext’w  R_Su=2'(w  RG_Avail'w  R_Ga) 

w  R_Su  :=  5 

wR_Ga:=-5 


Given 

w  RS_Ach_Ext'w  R_Su=2'(w  RG_Avail'w  R_Ga) 
w  R_Su  "*■  w  R_Ga=  1 


w  R_Su 
wR_Ga 


:-Find(wRSu,wRGa) 


w  r  Su  =0.788732394366197 
w  r  Ga  =  0.21 1267605633803 


Purpose:  Relate  Tech.  Sup./Be  Eff.  to  Tech.  Sup./Exp.  our  Fron. 
User  Rating  is  as  important  as  New  Warfighting  Spectrum 

w  TS  'w  T_Be'w  TBE_User=w  TS‘W  T_Ex'w  TEF_NWS 

Reduces  to: 

w  T_Be'w  TBE_User=w  T_Exw  TEF.NWS 
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Purpose:  Relate  Tech.  Sup./Be  Eff.  to  Tech.  Sup./Simplify  Operations 
User  Rating  is  twice  as  important  as  Enables 
w  T_Be'w  TBE_User=2'(w  T_SOw  TSO_Enables) 


Purpose:  Relate  Tech.  SupVExp.  our  Fron.  to  Tech.  Sup. /Win 
Warfare  Advantage  is  as  important  as  New  Warfighting  Spectrum. 
w  T_Ex'w  TEF_NWS=w  T_Wi  w  TSW_War 


Purpose:  Relate  Tech.  Sup./Exp.  our  Fron.  to  Tech.  Sup./Win 
Warfare  Advantage  is  three  times  more  important  then  AF  savings  Ratio. 
W  x_wi  -W  TSW_War=3'(w  T_AF'W  TA_Af) 
w  TA_AF= 1 


Purpose:  Relate  AF  Savings  Ratio  to  Programatic  Risk 
AF  Savings  Ratio  is  5  times  as  important  as  Programatic  Risk 
w  XAF  w  ta  af=5  w  xpr 

w  TA_AF=1 
WT_AF  :=1 
wT_Ex:=1 
wT_SO  :=1 
wT_Be:=1 
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WT_PR'-1 
wT_Wi :=1 


Given 

w  T_Be'w  TBE_User=2'(w  T_SO‘w  TSO_Enables) 
w  T_Ex'w  TEF_NWS=w  T_Wi  w  TSW_War 


w  T_Ex'w  TEF_NWS=w  T_Wi  w  TSW.War 
w  T_Wi " w  TSW_War=3 '  (w  T_Af) 
wT_AF=5wT_PR 


w  T_AF  +  w  T_Be  +  w  T_Ex+  w  T_PR  +  w  T_SO  +  w  T_Wi=  1 


WT_AF 

wT_Be 


wT_Ex 
w  T_PR 
wT_SO 
wT_Wi 


:  -  Find  (w  T_AF ,  w  T  Bc ,  w  T_Ex,  w  T  PR,  wT_SQ,w  T_wj ) 


WT_AF 
wT_Be 
w  T_Ex 
w  T_PR 
wT_SO 
wT_Wi 


0.04568107834005S 
0.23967599747762^ 
0.27408647004035^ 
0.00913621566801: 
0.0933 802587575 1 ( 
0.338039979716436 


Purpose:  Relate  Reach  to  Awareness 

Reach/Sustain/ Achieve/Extend  Life  is  three  times  as  important  as 
Awareness/Gain/ Availability. 
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w  RS_Ach_Ext'w  R_Su’w  Reach=3’(w  RG_Avail'w  A_Ga'w  Aware) 
wA_Ga=1 


Purpose:  Relate  Reach  to  Power 
Achieve/Extend  Life  is  as  important  as  standoff 
w  RS_Acw  R_Su‘w  Reach=w  PL_Stw  P_Lew  Power 
w  P_Le=  1 


Purpose:  Relate  Tech.  Sup.  to  Reach 

New  Warfighting  Spectrum  is  twice  as  important  as  Sustain/Extend  Life. 
2'  (w  RS_Ac  'w  R_Su'w  Reach) ®w  TS  w  T_Wi -w  TE_Wa 
w  Reach  -~2^ 
w  Aware  :=  25 

w  Power :=  25 
w  'pg  i  —  .25 

Given 

w  RS_Ach_Ext*w  R_Su‘w  Reach~^‘(w  RG„Avail'w  Aware) 
w  RS_Ach_Ext‘w  R_Su‘w  Reach- w  PL_survival  ‘w  Power 
2(w  RS_Ach_Ext‘w  R_Su'w  Reach)=w  TS’W  T_Wi”w  TSW^War 
w  Reach  w  Aware  w  Power  +  w  TS=  * 
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Reach  ’ w  Aware » w  Power » w 


0.28406892043272: 

0.0400097071032 

0.092022326337361 

0.583899046126716 


Appendix  H:  Air  Vehicles’  Programs  and  Scoring 
Description  of  Air  Vehicles  Programs  Scored 

The  following  information  was  provided  my  Mr.  Dave  Perez  of  Air  Vehicles.  This 
information  is  comes  from  program  charts,  internal  program  summary  sheets,  and  the 
FY98  Air  Vehicles  Technology  Area  Plan.  The  letters  by  each  program  represent  an 
arbitrary  coding  system  for  the  programs  to  make  references  to  them  simpler. 

A.  Weapons  Bay  Noise  Suppression  -  The  objective  of  this  effort  is  to 
develop  analysis  methods  for  the  prediction  of  flow-induced  acoustic 
levels  in  weapon  bays  and  unsteady  aerodynamic  loads  impinging  on 
structures.  Suppression  techniques  will  also  be  developed  for  both 
weapons  bay  acoustics  and  unsteady  aerodynamic  loads.  Current  passive 
suppression  systems  for  weapons  bays  are  only  effective  for  single  point 
designs.  The  active/adaptive  system  will  provide  acoustic  suppression  for 
all  bay  configurations  and  flight  conditions  resulting  in  significant  cost 
savings  by  eliminating  the  required  redesign  of  passive  systems. 

B.  Composite  Repair  of  Aircraft  Structures  -  Objectives:  Develop  and 
demonstrate  advanced  composite  repair  technology  for  components  of 
aging  aircraft.  Restore  structural  integrity,  extended  operational  life,  [and] 
reduce  operating  costs  of  aging  aircraft. 

C.  UCAV-ITAC  -  The  program  is  developing  the  flight  control  and  flight 
management  algorithms  for  cooperative  semi-autonomous  unmanned 
combat  air  vehicles  with  manned  aircraft  in  a  composite  tactical  strike 
package.  Mission  performance  improvements  resulting  from  cooperative 
flight  operations  of  a  multi-ship  strike  package  will  be  evaluated  and 
demonstrated.  The  adaptability  and  experience  of  the  pilot(s)  are 
integrated  with  automated  decision  aiding  processes,  advanced  multi-ship 
flight  path  control,  and  cooperative  planning  technologies  to  accomplish 
the  control  tasks  of  multi-ship  flight  management.  Air-to-Surface 
missions  are  being  addressed.  Management  and  supervision  of  the 
unmanned  vehicles  will  be  provided  by  a  pilot  in  another  aircraft  of  the 
strike  formation  or  an  operator  aboard  an  AWACS/JSTARS  aircraft. 

D.  Structurally  Integrated  Thermal  Energy  Management  (SITE-M)  -  The 
objective  of  this  program  is  to  develop  low  observable  compatible 
supportable  advanced  structural  concepts  for  thermal  energy  management 
(structurally  integrated  nozzles). 
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E.  Actuation  Health-Monitoring  Systems  -  This  program  will  develop  an 
intelligent  monitoring  system  capable  of  diagnosing  failures  in  actuation 
systems  on  air  vehicles.  This  information  will  result  in  faster  repair  cycles 
since  what  is  wrong  will  be  quickly  determined. 

F.  Space-Based  System-of-Sy stems  Advanced  Development  Demo  & 
Validation  Program  -  This  program  is  still  being  defined.  It  will  develop 
advanced  technologies  that  enable  an  unmanned  reusable  space  launch 
vehicle. 

The  program  was  difficult  to  score  under  “Reach”  because  it 
appeared  there  was  nothing  to  compare  the  program  to.  This  program  is 
extremely  new,  but  is  supposed  to  be  part  of  an  effort  that  will  develop  a 
completely  reusable  two-stage  launch  vehicle.  The  only  known  reusable 
launch  vehicle  is  the  Space  Shuttle,  which  is  not  completely  reusable.  The 
Space  Shuttle  is  capable  of  placing  55,000  pounds  into  low  Earth  orbit 
(NASA,  1988),  while  Air  Vehicles’  proposed  system  will  be  capable  of 
placing  6000  pounds  into  low  Earth  orbit.  This  large  mismatch  in  scale 
makes  the  Shuttle  a  poor  comparison. 

A  better  comparison  is  the  Delta  II 7320  (McDonnell  Douglas 
Corporation,  1996).  The  Delta  II 7320  vehicle  is  capable  of  placing  6030 
pounds  into  low  earth  orbit.  The  proposed  system  will  also  be  capable  of 
placing  6000  pounds  into  low  earth  orbit  (McDonnell  Douglas 
Corporation,  1996:  “Table  2-3.  Mission  Capabilities,”  page  2-6).  The 
Delta  II  takes  3  months  to  launch  and  is  available  only  once  during  that 
time.  Hence,  availability  is  1/60,  assuming  20  working  days  in  a  month.  It 
takes  3  months  to  erect,  prepare  and  fuel  the  Delta  II.  (McDonnell  Douglas 
Corporation,  1996:  “Typical  Mission  Plan”  schedule,  page  6-29). 

G.  More  Electric  Aircraft  Technology  Validation  (MTV)  -  In  a  joint  Air 
Force/NASA  effort,  the  More  Electric  Aircraft  technology  Validation 
(MTV)  flight  validation  program  will  demonstrate  that  electric  actuation  is 
a  mature  technology  for  high  horsepower  flight-critical  control 
applications.  An  electrohydrostatic  actuator  (EHA)  will  be  developed  and 
flown  on  a  NASA  F/A-18  aircraft.  AFRL’s  Propulsion  and  Power 
Directorate  will  provide  a  secondary  power  generation  and  distribution 
system  for  the  F/A-18.  NASA  will  provide  the  flight  test  aircraft  and 
support.  Successful  completion  of  this  program  will  put  power-by-wire 
actuation  “on  the  shelf’  for  military  and  commercial  applications.  Electric 
actuation  is  desirable  because  of  better  reliability,  easier  maintainability, 
and  potential  weight  savings. 
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Appendix  I:  Graphical  Sub-value  Contribution  to  each  Value 
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Figure  59:  Value  Contributions  for  “Support  the  Warfighter” 
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Figure  60:  Value  Contributions  to  “Reach” 
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Figure  61:  Value  Contributions  to  “Awareness’ 
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Figure  62:  Value  Contributions  to  Power 
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Figure  63:  Value  Contributions  to  Technological  Superiority 
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Figure  64:  Value  Contributions  to  “Reach/Gain” 
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Figure  65:  Value  Contributions  to  “Reach/Sustain’ 
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Figure  66:  Value  Contributions  of  “Be  Efficient”  under  Technological 

Superiority 
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Figure  67:  Value  Contributions  of  “Expand  our  Frontier”  under 
Technological  Superiority 
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Figure  68:  Value  Contributions  of  “Simplify  Operations”  Under 
Technological  Superiority 
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Figure  69:  Value  Contributions  of  “Win”  Under  Technological  Superiority 
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Appendix  J:  Sensitivity  Analysis  of  all  Values  and  Evaluation  Measures 
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Figure  70:  Sensitivity  of  Reach  (Left)  and  Awareness  (Right) 


Figure  71:  Sensitivity  of  Power  (Left)  and  Technological  Superiority  (Right) 


Figure  72:  Sensitivity  of  Reach  Gain  (Left)  and  Reach  Sustain  (Right) 


igure  74:  Sensitivity  of  Expand  our  Frontier  (Left)  and  Programmatic  Risk  (Right)  under  Technologica 

Superiority 


re  77:  Sensitivity  of  Regeneration  (Left)  and  Response  Time  (Right)  under  Reach  under  Gain 


Figure  78:  Sensitivity  of  Payload-Range  under  Reach  under  Gain 
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Figure  83:  Sensitivity  of  Regeneration  under  Awareness  under  Gain 


B5:  Sensitivity  of  Dual  Use  (Left)  and  Intermediate  Products  (Right)  under  Technological  Superiority  under 

Be  Efficient 


Figure  86:  Sensitivity  of  Multiuse  (Left)  and  Uniqueness  (Right)  under  Technological  Superiority  under  Be 

Efficient 


Figure  87:  Sensitivity  of  User  Rating  under  Technological  Superiority  under  Be  Efficient 
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Figure  89:  Sensitivity  of  Enables  (Left)  and  Interoperability  (Right)  under  Technological  Superiority  under 

Simplify  Operations 


Figure  90:  Sensitivity  of  Ratio  of  People  under  Technological  Superiority  under  Simplify  Operations 


:  Sensitivity  of  Cost  to  Adversary  (Left)  and  Lead  (Right)  under  Technological  Superiority  under  Win 


Figure  92:  Sensitivity  of  Warfare  Advantage  under  Technological  Superiority  under  Win 


Appendix  K:  Intensity  of  Change  in  Warfighter  Support  due  to  Technology 

Intuitively,  the  change  in  warfighter  technological  intensity  has  a  simple  R&D 
interpretation.  It  is  based  on  the  idea  that  R&D  should  always  improve  over  the  current 
deployed  state  of  the  art.  It  is  illogical  to  accomplish  R&D  to  achieve  the  current  existing 
deployed  state  of  the  art  or  to  improve  to  some  level  below  the  current  existing  deployed 
state  of  the  art.  Therefore,  the  change  in  warfighter  technological  intensity  is  always  a 
positive  value,  indicating  an  improvement  in  the  deployed  state  of  the  art  for  technology. 

At  first,  it  was  thought  this  parameter  would  be  measured  on  a  ratio  scale,  since 
the  change  in  warfighter  technological  support  is  on  a  ratio  scale  and  cost  is  also  on  a 
ratio  scale.  However,  the  parameter  is  not  measured  on  a  common  ratio  scale  because  for 
each  program,  change  in  warfighter  technological  support  is  divided  by  a  different  cost. 
This  fails  the  allowable  transformations  for  a  ratio  scale,  which  is  multiplication  by  a 
scalar  (Luce,  Bush  and  Galanter,  1963:  8-13).  The  parameter  would  be  only  be  on  a  ratio 
scale  if  change  in  warfighter  technological  support  was  divided  by  the  same  cost  for 
every  program. 
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